Mobile systems and methods of supporting natural language human-machine interactions

ABSTRACT

A mobile system is provided that includes speech-based and non-speech-based interfaces for telematics applications. The mobile system identifies and uses context, prior information, domain knowledge, and user specific profile data to achieve a natural environment for users that submit requests and/or commands in multiple domains. The invention creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context and presenting the expected results for a particular question or command. The invention may organize domain specific behavior and information into agents, that are distributable or updateable over a wide area network.

FIELD OF THE INVENTION

The invention provides mobile devices having a speech interface and/or acombination of a speech interface and a non-speech interface to enablenatural language human-machine interactions. More specifically, theinvention enables mobile users to submit natural language speech and/ornon-speech questions or commands in a wide range of domains. The mobiledevice is configured to present responses in a natural manner.

BACKGROUND OF THE INVENTION

Telematic systems are systems that bring human-computer interfaces tomobile environments. Conventional computer interfaces use somecombination of keyboards, keypads, point and click techniques and touchscreen displays. These conventional interface techniques are generallynot suitable for a mobile environments, due at least in part to thespeed of interaction and the inherent danger and distraction. Therefore,speech interfaces are being adopted in many telematic applications.

However, creating a natural language speech interface that is suitablefor use in the mobile environment has proved difficult. Ageneral-purpose telematics system should accommodate commands andrequests from a wide range of domains and from many users with diversepreferences and needs. Further, multiple mobile users may want to usesuch systems, often simultaneously. Finally, most mobile environmentsare relatively noisy, making speech recognition inherently difficult.

Retrieval of both local and network hosted online information andprocessing of commands in a natural manner remains a difficult problemin any environment, especially a mobile environment. Cognitive researchon human interaction shows that verbal communication, such as a personasking a question or giving a command, typically relies heavily oncontext and domain knowledge of the target person. By contrast,machine-based requests (a request may be a question, a command, and/orother types of communications) may be highly structured and may not beinherently natural to the human user. Thus, verbal communications andmachine processing of requests that are extracted from the verbalcommunications may be fundamentally incompatible. Yet the ability toallow a person to make natural language speech-based requests remains adesirable goal.

Research has been performed on multiple fields of natural languageprocessing and speech recognition. Speech recognition has steadilyimproved in accuracy and today is successfully used in a wide range ofapplications. Natural language processing has previously been applied tothe parsing of speech queries. Yet, a limited number of systems havebeen developed that provide a complete environment for users to makenatural language speech requests and/or commands and receive naturalsounding responses in a mobile environment. There remain a number ofsignificant barriers to creation of a complete natural language verbaland/or textual-based query and response environment.

The fact that most natural language requests and commands are incompletein their definition is a significant barrier to natural languagequery-response interaction. Further, some questions can only beinterpreted in the context of previous questions, knowledge of thedomain, or the user's history of interests and preferences. Thus, somenatural language questions and commands may not be easily transformed tomachine processable form. Compounding this problem, many naturallanguage questions may be ambiguous or subjective. In these cases, theformation of a machine processable query and returning of a naturallanguage response is difficult at best.

Even once a question is asked, parsed and interpreted, machineprocessable requests and commands must be formulated. Depending on thenature of the question, there may not be a simple set of requests thatreturn an adequate response. Several requests may need to be initiatedand even these requests may need to be chained or concatenated toachieve a complete result. Further, no single available source mayinclude the entire set of results required. Thus, multiple requests,perhaps with several parts, may need to be made to multiple datasources, which can be located both locally or remotely. Not all of thesesources and requests may return useful results or any results at all.

In a mobile environment, the use of wireless communications may furtherreduce the chances that requests will be complete or that successfulresults will be returned. Useful results that are returned are oftenembedded in other information and may need to be extracted therefrom.For example, a few key words or numbers often need to be “scraped” froma larger amount of other information in a text string, table, list,page, or other information. At the same time, other extraneousinformation such as graphics or pictures may need to be removed toprocess the response in speech. In any case, the multiple results shouldbe evaluated and combined to form the best possible answer, even in thecase where some requests do not return useful results or fail entirely.In cases where the question is ambiguous or the result inherentlysubjective, determining the best result to present is a complex process.Finally, to maintain a natural interaction, responses should be returnedrapidly to the user. Managing and evaluating complex and uncertainrequests, while maintaining real-time performance, is a significantchallenge.

These and other drawbacks exist in existing systems.

SUMMARY OF THE INVENTION

The invention overcomes these and other drawbacks of prior telematicsystems.

According to one aspect of the invention, speech-based andnon-speech-based systems are provided that act on commands and retrieveinformation. The invention uses context, prior information, domainknowledge, and user specific profile data to achieve a naturalenvironment for users that submit requests and/or commands in multipledomains. At each step in the process, accommodation may be made for fullor partial failure and graceful recovery. The robustness to partialfailure may be achieved through the use of probabilistic and fuzzyreasoning at several stages of the process. This robustness to partialfailure promotes the feeling of a natural response to questions andcommands.

According to another aspect of the invention, the interactive naturallanguage system (herein “the system”) may be incorporated into mobiledevices or may be connected to the mobile device via a wired or wirelessconnection. The mobile device may interface with computers or otherelectronic control systems through wired or wireless links. The mobiledevice may also operate independently of a mobile structure and may beused to remotely control devices through a wireless local areaconnection, a wide area wireless connection or through othercommunication links.

According to one aspect of the invention, software may be installed ontothe mobile device that includes an input module that captures the userinput; a parser that parses the input, a text to speech engine modulefor converting text to speech; a network interface for enabling thedevice to interface with one or more networks; a non-speech interfacemodule, an event manager for managing events and/or other modules. Insome embodiments, the event manager may be in communication with acontext description grammar, a user profile module that enables userprofiles to be created, modified and/or accessed, a personality modulethat enables various personalities to be created and/or used, an agentmodule, an update manager and one or more databases. It will beunderstood that this software may be distributed in any way between amobile device, a computer attached to a mobile structure, a desktopcomputer or a server without altering the function, features, scope, orintent of the invention.

According to one aspect of the invention, the system may include aspeech unit interface device that receives spoken natural languagerequests, commands and/or other utterances from users, and a computerdevice or system that receives input from the speech unit and processesthe input and responds to the user with a natural language speechresponse.

According to another aspect of the invention, the system may beinterfaced by wired or wireless connections to one or more othersystems. The other systems may themselves be distributed betweenelectronic controls or computers that are attached to a mobile structureor are located external to the mobile structure. The other systems mayinclude, electronic control systems, entertainment devices, navigationequipment, measurement equipment or sensors, or other systems. Externalsystems may also be provided with features that include payment systems,emergency assistance networks, remote ordering systems, automated orattended customer service functions, or other features.

According to another aspect of the invention, the system may be deployedin a network of devices that share a common base of agents, data,information, user profiles, histories or other components. Each user mayinteract with, and receive, the same services and applications at anylocation equipped with the mobile device on the network. For example,multiple mobile devices may be placed at different locations throughouta home, place of business, vehicle or other location. In such a case,the system can use the location of the particular device addressed bythe user as part of the context for the questions asked.

According to one embodiment of the invention, processing may beperformed at the mobile devices. The commands may be processed on-boardto enable the mobile devices to control themselves and/or to controlother mobile devices, fixed computers, mobile telephones, and otherdevices. Additionally, mobile devices may track context.

According to one embodiment of the invention, infrastructure may beprovided to maintain context information during multimodal interactions,such as speech and/or non-speech interactions. According to oneexemplary embodiment of the invention, context information may bemaintained in a multimodal environment by providing communicationchannels between mobile device, or multimodal devices, and the system.The communication channels allow the system to receive multimodal inputsuch as text-based commands and questions and/or voice-based commandsand questions. According to another embodiment of the invention, themultimodal input may include a string of text, such as keywords, thatare received as commands or questions. According to yet anotherembodiment of the invention, the system may synchronize the contextbetween the mobile devices and the speech-based units. In order to senda response to the corresponding mobile device, the system may track thesource and send the response to the corresponding speech interface orthe non-speech interface.

According to an alternative embodiment of the invention, contextinformation may be maintained using a context manager that may becentrally positioned to receive input from multiple mobile devices andto provide output to multiple mobile devices. According to oneembodiment, the mobile devices that communicate with the context managermay register through a registration module and may subscribe to one ormore events. According to another embodiment of the invention, thecontext manager may receive input in Context XML form, for example. Theother registered mobile devices may be informed of context changesthrough a context tracking module to enable synchronizing of contextacross the registered modules. According to one embodiment of theinvention, registered modules may be added or removed from the system.The registered modules may include dynamic link libraries (DLLs), orother information sources, that are specific to multimodal devices.

According to yet another alternative embodiment of the invention,context information may be determined from a command or request that ispresented in a textual format and/or a command or request that ispresented as an utterance and processed using a multi-pass automaticspeech recognition module that transcribes the utterance to a textmessage. The command or request may be compared against a contextdescription grammar to identify a match. Any active grammars in thecontext description grammar may be scored against the command or requestand a best match may be sent to a response generator module. Agents maybe associated with corresponding response generator modules and mayretrieve the requested information for generation of a response. Theagents may update a context stack to enable follow-up requests.

According to another embodiment of the invention, mobile devices may beconfigured to allow verbal annotations of objects stored thereon. Mobiledevices may transcribe the verbal annotation to text and store thetextual annotation with the object. Alternatively, mobile devices may beconfigured to enable users to manually input textual descriptions thatare stored along with the objects. According to one embodiment of theinvention, the textual annotations and/or textual descriptions may beclassified and searched. In an alternative embodiment, mobile devicesmay classify and search the verbal annotations rather than textualannotations. However, classifying and searching verbal annotations maybe considerably more difficult than classifying and searching textualannotations and/or textual descriptions.

According to one embodiment of the invention, the textual annotationsand textual descriptions may be communicated using a short messageservice on the mobile telephone or other device. Short message serviceis a text message service that enables sending and receiving of shorttextual messages. The textual messages may be stored at data centers forforwarding to intended recipients. Other configurations may be used.

According to another aspect of the invention, domain specific behaviorand information may be organized into data managers. Data managers areautonomous executables that receive, process and respond to userquestions, queries and commands. The data managers provide complete,convenient and re-distributable packages or modules of functionalitythat are typically directed to a specific domain of applications. Datamanagers may be complete packages of executable code, scripts, links toinformation, and other forms of communication data that provide aspecific package of functionality, usually in a specific domain. Inother words, data managers may include components for extending thefunctionality to a new domain. Further, data managers and theirassociated data may be updated remotely over a network as new behavioris added or new information becomes available. Data managers may usesystem resources and the services of other, typically more specialized,data managers. Data managers may be distributed and redistributed in anumber of ways including on removable storage media, transfer overnetworks or attached to emails and other messages. An update manager maybe used to add new data managers to the system or update existing datamanagers.

In order to enhance the natural query and response environment, thesystem may format results to increase understandability to users.Formatting and presentation of results may be based on the context ofthe questions, the contents of the response being presented, the historyof the interaction with the user, the user's preferences and interestsand the nature of the domain. By contrast, rigid, highly formatted, orstructured presentation of results may be deemed unnatural by manyusers.

According to another embodiment of the invention, the system maysimulate some aspects of a human “personality”. In some cases, thepresentation of the response and the terms that are used to provide theresponse may be randomized to avoid the appearance of rigidly formattedor mechanical responses. The use of other simulated personalitycharacteristics is also desirable. For example, a response that may beupsetting to the user may be presented in a sympathetic manner.Furthermore, results of requests may be long text strings, lists, tablesor other lengthy sets of data. Natural presentation of this type ofinformation presents particular challenges because simply reading thelong response is generally not preferred. Instead, the system may parseimportant sections from the response and may initially provide onlyreports. Determining what parts of a long response are presented may bebased on the context of the questions, the contents of the responsebeing presented, the history of the interaction with the user, theuser's preferences and interests and the nature of the domain. At thesame time, the system may give the user interactive control over whatinformation to present and how much information to present, to stop theresponse all together, or to take other actions.

The invention may be applied as a user interface to telematics systemsin a wide variety of environments. These environments may include, butare not limited to, the following: 1) personal automobiles, rentedautomobiles, or fleet automobiles; 2) motorcycles, scooters, and othertwo wheeled or open-air vehicles; 3) commercial long-haul and short haultrucks; 4) delivery service vehicles; 5) fleet service vehicles; 6)industrial vehicles; 7) agricultural and construction machinery; 8)water-borne vehicles; 9) aircraft; and, 10) specialized military, lawenforcement and emergency vehicles.

According to another aspect of the invention, the system may process andrespond to questions, requests and/or commands. Keywords or context maybe used to determine whether the received utterance and/or textualmessage includes a request or command. For example, utterances mayinclude aspects of questions, requests and/or commands. For example, auser may utter “tune in my favorite radio station.” A request isprocessed to determine the name, the channel, and time for the usersfavorite radio station. If the programming on that station is of a typethe user generally does not listen to, the system can suggest using analterative option, such as listening to a CD more likely to please theuser. A command must be executed to set a tune the radio station.

The invention can be used for generalized local or network informationquery, retrieval and presentation in a mobile environment. For each userutterance, including a question or query or set of questions or queries,the system may perform multiple steps possibly including: 1) capturingthe user's question or query through speech recognition operating in avariety of real-world environments; 2) parsing and interpreting thequestion or query; 3) determining the domain of expertise required andthe context to invoke the proper resources, including agents; 4)formulating one or more queries to one or more local and/or network datasources or sending appropriate commands to local or remote devices orthe system itself; 5) performing required formatting, variablesubstitutions and transformations to modify the queries to a form mostlikely to yield desired results from the available sources; 6) executingthe multiple queries or commands in an asynchronous manner and dealinggracefully with failures; 7) extracting or scraping the desiredinformation from the one or more results, which may be returned in anyone of a number of different formats; 8) evaluating and interpreting theresults, including processing of errors, gathered and combine them intoa single best result judged to be “best” even if the results areambiguous, incomplete, or conflicting; 9) performing requiredformatting, variable substitutions and transformations to modify theresults to a form most easily understood by the user; 10) presenting thecompound result, through a text to speech engine or multimodalinterface, to the user in a useful and/or expected manner; 11)optionally, providing a response to users indicating the success orfailure of the command, and possibly including state information; orother steps.

The above steps may be performed with knowledge of the domain ofexpertise, the context for the question or command, domain specificinformation, the history of the user's interactions, user preferences,available information sources or commands, and responses obtained fromthe sources.

Probabilistic or fuzzy set decision and matching methods may be appliedto deal with inconsistent, ambiguous, conflicting and incompleteinformation or responses. In addition, asynchronous queries may be usedto provide rapid and graceful failure of requests or commands that allowthe system to robustly return results quickly and in a manner that seemsnatural to the user.

Many everyday questions are inherently subjective and result in answersthat are a matter of opinion or consensus, as much as fact. Suchquestions are often ad hoc in their nature, as well. The system,according to another aspect of the invention, may use adaptive,probabilistic, and fuzzy set decision and matching methods to identifythe subjective nature of the question and to evaluate a range ofpossible answers, wherein one or more answers may be selected that mostaccurately represent the type of result desired by the user.

The context and expected results from a particular question may behighly dependent on the individual asking the question. Therefore, thesystem may create, store and use personal profile information for eachuser. Information in the profile may be added and updated automaticallyas the user uses the system or may be manually added or updated by theuser or by others. Domain specific agents may collect, store and usespecific profile information, as may be required for optimal operations.Users may create commands for regularly used reports, automaticallygenerated alerts, and other requests and for the formatting andpresentation of results. The system may use profile data in interpretingquestions, formulating requests, interpreting request results andpresenting answers to the user. Examples of information in a userprofile includes, history of questions asked, session histories,formatting and presentation preferences, special word spelling, terms ofinterest, special data sources of interest, age, sex, education,location or address, place of business, type of business, investments,hobbies, sports interests, news interests and other profile data.

According to one aspect of the invention, the system may attempt toprovide rapid responses in order to provide a natural question andresponse environment. The rapid responses may be provided withoutobtaining additional information. The system may determine agentcomposition, context and/or domain for a user's question or command, forexample, by using a real-time scoring system or other technique. Basedon this determination, the system may trigger one or more agents torespond to the user's question or command. The agents may make one ormore requests and rapidly return a formatted response. Thus, users mayreceive direct responses to a set of questions, each with a differentresponse or context. In some cases, the available information, includingthe request results, may not adequately answer the questions presented.In such situations, the user may be asked one or more follow-upquestions to resolve the ambiguity. Additional requests may then be madebefore an adequate response is provided. In these cases, the system mayuse context information, user profile information and/or domain specificinformation to minimize the interaction with the user required todeliver a response.

If the confidence level of the domain or context score is not highenough to ensure a reliable response, the system may request that theuser verify that the question or command is correctly understood. Ingeneral, the question may be phrased to indicate the context of thequestion including all criteria or parameters. If the user confirms thatthe question is correct, the system may proceed to produce a response.Otherwise, either the user can rephrase the original question, perhapsadding additional information to remove ambiguity, or the system may askone or more questions to attempt to resolve the ambiguity or otheractions may taken.

According to one aspect of the invention, the system may accept anynatural language question or command and, as a result, may be subject toambiguous requests. To assist users in formulating concise questions andcommands, the system may support a voice query language. The languagemay help users clearly specify the keywords or contexts of the questionor command along with the parameters or criteria. The system may providebuilt in training capabilities to help the user learn the best methodsto formulate their questions and commands.

In order to make the responses to user's questions and commands seemmore natural, the system may employ one or more dynamically invokeablepersonalities and/or emotional models. Personalities and emotionalmodels have specific characteristics that simulate the behavioralcharacteristics of real humans. Examples of these characteristicsinclude, sympathy, irritation, and helpfulness and associated emotions.The personality also randomizes aspects of responses, just as a realhuman would do. This behavior includes randomization of terms used andthe order of presentation of information. Characteristics of thepersonality and/or emotions are invoked using probabilistic or fuzzy setdecision and matching methods and using criteria, including the contextfor the question, the history of the user's interaction, userpreferences, information sources available, responses obtained from thesources.

According to another aspect of the invention, special procedures may beemployed to present information in the form of long text strings,tables, lists or other long response sets. Simply presenting a long setof information in an ordered manner may not be considered natural orwhat most users have in mind. The system may use probabilistic or fuzzyset matching methods to extract relevant information and present thesesubsets first. Further the system may provide commands allowing users toskip through the list, find keywords or key information in the list orstop processing the list altogether.

According to one embodiment of the invention, the system may supportmultiple users that access the system at different times. According toanother embodiment of the invention, the system may support multipleusers that access the system during a same session, in an interleaved oroverlapping manner. The system may recognize the multiple users by name,voice, or other characteristic and may invoke a correct profile for eachuser. If multiple users are addressing the system in overlapping orinterleaved sessions, the system may identify the multiple users and mayinvoke one or more corresponding profiles. For applications that requiresecurity safeguards, the multiple users may be verified using voiceprintmatching, password or pass-phrase matching, or other securitysafeguards.

When multiple users are engaged in interleaved sessions, the system maygracefully resolve conflicts using a probabilistic or fuzzy set decisionmethod for each user. This process may simulate the manner in which ahuman would address multiple questions from various sources. Forexample, the system may answer short questions first in time whileanswering longer questions later in time. Alternatively, the system mayanswer questions in the order that they are received, among otherconfigurations.

Since the invention may operate in many environments, including mobileenvironments with background noise, point noise sources and peopleholding conversations, filtering of speech input may be advantageous.The invention may use, for example, either one-dimensional ortwo-dimensional array microphones (or other devices) to receive humanspeech. The array microphones may be fixed or employ dynamic beamforming techniques. The array pattern may be adjusted to maximize gainin the direction of the user and to null point noise sources.Alternatively, microphones may be placed at particular locations withinthe mobile environment near where occupants are likely to use thesystem. These microphones can be single microphones, directionalmicrophones or an array of microphones. Speech received at themicrophones may then be processed with analog or digital filters tooptimize the bandwidth, cancel echoes, notch-out narrow band noisesources, or perform other functions. Following filtering, the system mayuse variable rate sampling to maximize the fidelity of the encodedspeech, while minimizing required bandwidth. This procedure may beparticularly useful in cases where the encoded speech is transmittedover a wireless network or link.

The invention can be applied to a wide range of telematics applications.General applications areas can include, but are not limited to remote orlocal vehicle control, information query, retrieval and presentationfrom local or network sources, safety applications, and securityapplications.

The system may provide local or remote control functions for the system,including devices that are located on the mobile structure or off themobile structure. Users may initiate commands locally or remotely.Typically, remote operation may be conducted through an IP connection, atelephone connection, or other connections. The user may address spokencommands to a mobile device or desktop unit, which may send the commandsto controllers on the vehicle over wireless links. Other remote commandtechniques may be used. The system may process commands in a nearlyidentical manner to a request. One difference being that the result ofthe command is generally an action rather than a response. In manycases, the system may give the user a cue or response to indicate thatthe command has been successfully executed or has failed. In cases offailure, an interactive session may be started to allow the user toresolve the difficulty or formulate a command more likely to succeed.

The invention provides users, including mobile structure operators, withthe capability to control most any mobile system functions usinginteractive speech and non-speech commands and/or requests. Generally,controls of a critical nature, or having safety implications, may employfail-safe checks that, before execution, verify that a command will notcreate a hazardous condition. Manual overrides may also be provided asan extra precaution. The invention may provide built in help and userguidance for the devices under control. This guidance may includestep-by-step training for operators that are learning to use thefeatures of the mobile structure. The system can provide extensiveinteractive guidance when commands cannot be executed or when commandsfail. This advice may include suggestions to reformulate the command tosucceed, suggestions to work around a failure, suggestions foralternative commands that may achieve a similar function, or othersuggestions. Examples of control functions that can be performed fromlocal or remote locations by the invention include:

-   -   1. Control of mobile structure multimedia entertainment        electronics, such as radio, CD player, or video player. This        control can be based on user specified play lists and may be        sensitive to user profile information including, preferences        history, or other information. The invention may include the        ability to control multiple or individual multimedia        entertainment stations.    -   2. Control of communications devices such as cell phones, voice        mail systems, fax systems, text or instant messaging systems,        call and message forwarding systems, email systems and other        communication devices. This control includes controlling        features such as, address books, phone books, call forwarding,        conference calling, and voice mail, among others.    -   3. Local or remote control of mobile structure systems. Most any        device on the mobile structure may be controlled including, door        locks, window controls, interior temperature controls, shifting        of the transmission, turn signals, lights, safety equipment,        engine ignition, cruse control, fuel tank switches, seat        adjustments, specialized equipment such as winches, lifting        systems or loading systems, and other mobile structure systems.    -   4. Systems that are located external to the mobile structure        typically may be controlled through wireless links including,        garage door openers, gate controllers, mobile structure entry        security passes, automated toll collection systems, and mobile        structure weighing systems and other external systems.    -   5. Mobile structure power management and systems control. The        invention may provide the mobile structure operator with        information on limitations and on tips for better power        management or fuel utilization or other systems control.    -   6. Diagnostic information management. The invention may provide        diagnostic information announcements and warnings for the mobile        structure operator. These announcements and warnings may be        interactive allowing the operator to request additional        information or to suggest various courses of action. The        invention may mediate a solution to the problem, including        providing access to a scheduling service, summoning help or        providing instructions for remedial action until a permanent        solution can be achieved. The system can ask the operator to        authorize the ordering of needed parts and may provide cost        estimates. The system may receive data for these announcements        and warnings from a wide range of sources including sensors and        vehicle control computers. Sensors may include fuel level        sensors, coolant temperature sensors, oil temperature sensors,        axel temperature sensors, tire pressure sensors, and other        sensors.    -   7. System status inquiry. Mobile structure operators may use the        interactive natural language interface of the invention to query        and receive reports on the status of systems that are located on        the mobile structure including, fuel level, interior        temperature, outside temperature, engine or other mobile        structure system status. The operator can further query the        system to receive more information or determine a course of        action if problem is detected.    -   8. Vehicle service history. The invention can provide the        vehicle operator or other personnel with interactive access to        the vehicle service history. The invention can provide        announcements or warnings as the time for service draws near.        The user can interact with the system to schedule the required        service, order required parts, receive cost estimates, or update        the service history. Users may customize the nature of this        interaction to suit their desires or policies.    -   9. Diagnostic and service history. The invention may provide        diagnostic and service history information to service personnel.        This information may include vehicle fault codes and other        information for devices that are under control of or measured by        the system. Alternatively, the invention may receive information        on the state and history of vehicle operation from other control        computers. The invention may provide interactive service        information and history. The service history may be queried and        presented through the speech interface or the non-speech        interface. The system may prompt service personnel for more        information if the record is deemed incomplete. In other cases,        the invention may prompt service personnel for information on        their actions if a change is detected in system status, such as        replacement of a part.

The invention can provide users or operators of a mobile structure withspecialized safety functions through the interactive speech interfaceand/or non-speech interface. The invention may use a dynamicallyevocable personality that is capable of creating announcements that areappropriate for the severity of the situation. The announcements andpersonalities may be under user control and configuration. Some examplesof these safety applications can include:

-   -   1. The invention can provide automated detection and reporting        of accident situations through the wireless communications link.        Information on an accident situation can be gathered from airbag        control systems or other sensors. Once an accident situation has        been detected the invention may use the interactive speech        interface and/or non-speech interface to determine the nature of        the accident and/or the condition of victims. This information,        along with location information and other relevant information,        may be reported over the wireless link. Alternatively, the        invention may establish voice channel communications or other        communications between occupants of the vehicle and emergency        personnel.    -   2. The invention can be used to store and retrieve medical        information on vehicle occupants. Following an accident,        emergency personnel may query the system for this information.        Alternatively, the system may annunciate a warning to emergency        personnel if a person has a special medical condition. The        system maintains the security of medical information through a        number of techniques, including not annunciating medical        information unless an accident is detected, or not annunciating        medical information unless that person or another authorized        person gives permission.    -   3. Occupants of the vehicle can summon help in the event of a        crime using the speech interface and/or non-speech interface.        Typical crimes may include robberies and hijackings. The        invention allows vehicle occupants to set panic or emergency        words or phrases that indicate to the system a crime is        occurring.    -   4. The invention can provide the vehicle operator with safety        announcements if an unsafe or potentially unsafe situation is        detected. The operator can use the interactive speech interface        and/or non-speech interface to obtain more information on the        situation or dismiss the alert. The operator can annunciate or        otherwise provide commands to remedy or mitigate the situation        during this dialog. Conditions that can be announced include,        following another vehicle too closely, too great a speed for the        road or conditions, obstruction on roadway, a fire in some part        of the vehicle, high cargo pressure or temperature, leaks, and        other information.    -   5. The interactive speech interface and non-speech interface of        the invention can provide the operator with real-time        assistance. This assistance may include, aid in parking or        backing a vehicle, aid with complex maneuvers, aid with optimal        operation of the vehicle, and other operations. The operator can        ask the system for advice or assistance with a planned maneuver        or operation. Alternatively, the invention can proactively offer        assistance if certain situations are detected.    -   6. The interactive speech interface and non-speech interface may        be used to improve vehicle security. Voiceprints or voice        authentication can be used to gain access to the vehicle or        start the vehicle. Alternatively or in addition, a password or        pass-phrase can be used. In another alternative speech security        can be used as a supplement to other vehicle security        techniques.    -   7. The invention can provide measurement of operator fatigue and        alert the operator or remote personnel if unacceptable levels of        fatigue are detected. The interactive speech interface and/or        non-speech interface can be used to query the operator to detect        fatigue. Alternatively, or in addition, other measurements of        operator fatigue can be used. If a fatigue situation is detected        the invention may initiate a dialog with the operator to        determine the extent of the problem and if required, ask the        operator to cease operation.

The invention can offer vehicle operators and occupants a variety ofservices that are useful while in the vehicle and/or while arriving at adestination. Further, users can employ the interactive natural languageinterface to customize these services to suit each individual. Someexamples of services that can be supported by the natural-languageinteractive speech interface of the invention, include:

-   -   1. Providing vehicle operators with interactive directions to a        destination or waypoint, wherein the user can specify a desired        destination and any preferred waypoints. A destination can be        specified in any manner including providing the name of a place,        an address, a name of a person, a name of a business, or other        type of information. As the trip progresses the system may        provide the operator with continued directions and warnings if a        mistake has been made. The operator can query the system for        additional information or request less information. Generally,        the system is interfaced with one or more navigation sensors and        local or remote map databases. The invention may provide        operators or passengers with alerts of upcoming points or        interest, exits or stops, hazards, or other conditions. The        users can query the system for more specific information.        Alternatively, the invention can provide operators and occupants        of the vehicle with an interactive guided tour. The system's        information query, retrieval and presentation capability can be        employed by users to receive additional information or points or        items of interest during the tour and may take into account        stored personal profile information for a user.    -   2. The invention can provide the operator of a vehicle with        interactive dynamic routing information. The routing can be        updated based on traffic conditions, weather conditions,        facilities availability, and information provided by the        operator. Generally, the system is interfaced with one or more        navigation sensors, local or remote map databases, and sources        of traffic, weather, and facilities use data.    -   3. The direction, routing and communications capabilities of the        invention can be combined in an interactive system that helps        one or more operators rendezvous at a predetermined destination        or any other convenient midpoint. The operators may use the        interactive natural language interface to communicate with the        system to arrange the rendezvous, to receive directions as they        travel to the rendezvous point and/or to communicate with the        other operators.    -   4. The navigational capabilities of the invention may be used to        place limits on where a vehicle is allowed to go and/or for how        long. The system may employ the interactive natural language        speech interface and/or non-speech interface to inform the        operator when the vehicle is approaching or has exceeded a        limit. The operator can query the system to determine the best        course of action to return to limits or prevent exceeding them.        Alternatively, the system can query the operator to determine        why they are exceeding the limits or to mediate a negotiation to        extend the limits if this is required by circumstances. This        capability is useful in several situations including, keeping a        delivery or passenger vehicle on a regular route, setting and        enforcing use limits on teenagers, preventing an operator from        using the vehicle in an unauthorized manner, or other        situations.    -   5. The interactive natural language interface of the invention        can be used to provide Customer Relationship Management (CRM)        services to vehicle operators and passengers. The user can        interact with the services offered via data networks, video        signals, or audio. The interaction can be performed with        automated services or a live Customer Service Representative.        Interactions with the customer service representatives can be        through any combination of possible techniques, such as, live        audio, live video, electronic messaging or email, instant        messaging, and other techniques. These services can be offered        by a number of entities including, vehicle manufacturers,        vehicle dealers, vehicle service organizations, automobile or        travel clubs, wireless carriers, travel service organizations,        or other organizations. The services offered can be personalized        to the occupants of the vehicle using a variety of information        including, user profile information, history, location, paths        traveled, time of day, day of week, or other information. In        addition, the system can offer customized services based on        information about the vehicle, including paths traveled,        distance, service history, type of equipment on vehicle. These        services can be accessed while a person is an occupant in a        vehicle, while the person is not an occupant but are using a        wireless or wired network equipped mobile device or while using        a wired or wireless network desktop system, among other        situations. Examples of these services include:        -   a. Location based marketing programs wherein occupants of            the vehicle receive promotional offers from merchants along            a route of travel. Occupants can query the system for offers            and promotions for goods, services, or other information,            along the travel route. The system may apply other available            information to form a response, including, the user profile,            history, location, or other information. The system can            provide optimized interactive routing assistance to the            vehicle operator. Alternatively, the system can provide            interactive offers and promotions for goods and services            along the route, or in advance of a particular trip.            Promotions can be offered for goods and services can include            but are not limited to, travel services, groceries, prepared            foods, vehicle service, fuel, entertainment, or other goods            and services.        -   b. Remote ordering and payment for goods and services. The            system can interactively present the menu or product catalog            using the list and table presentation capabilities of the            invention. The system may facilitate remote ordering by            using location information, customer preferences, customer            order history, etc. The system can manage a secure payment            wallet for the users. Voiceprints, spoken passwords, and            non-speech security methods (i.e. PIN pad, etc.) can be            combined to create the appropriate level of security.        -   c. Travel services for occupants of the vehicle. These            services can include, directories of travel and            entertainment services, or reservations for entertainment            restaurants, hotels and other accommodation. The system may            present directories, lists and menus using its interactive            list and table presentation capabilities. The travel service            capability can be used in conjunction with the remote            ordering and payment capabilities and the dynamic            interactive routing capability.        -   d. Answer specialized travel related questions in areas such            as vehicle registration, taxes, safety laws, required            inspections, weight limits, insurance coverage requirements,            insurance policy provisions, or other areas.    -   6. The invention can provide an operator or other occupant of a        vehicle with an interactive location sensitive shopping list or        a location and time sensitive task reminder list using the        natural language interface. Users can create the list while in        the vehicle, while on foot using a mobile device, or at a fixed        location using a handheld or desktop device. A user may grant        permission to other users to add tasks or shopping items to        their lists. Once in the vehicle, the system may provide        occupants with routing assistance to optimize travel time and        may provide reminders of items to be purchased and tasks to be        completed as the vehicle comes in close proximity to a        particular location, type of merchant or other service provider,        or when a set time has been reached, among other criteria.    -   7. Automatic interactive dispatch and reporting for fleet        vehicles having vehicle operators or other vehicle occupants        that use the speech interface and/or non-speech interface to        interact with these services. These services can include,        dynamic optimal routing, inventory of parts and other materials,        ordering of required parts and materials, work orders, receipt        generation, payments, or other services.    -   8. Sales force automation, sales reporting, contact database        management, calendar management, and call routing. The system        may employ its interactive list and table presentation        capabilities to supply catalog and pricing information or other        information. These services can use local or network data.        Additional services can include, memos, reminders, activity        lists, or other information.

Vehicle operators and other occupants can use the interactive naturallanguage interactive interface of the invention to perform many types ofinformation query, retrieval and presentation operations. Using thenatural language interactive interface, users can modify the parametersof queries or specify the presentation formats for results. Data used tocreate a response can be from any combination of local and remote datasources. User specific data can be synchronized between systems fixed toone or more vehicles, mobile structures and desktop systems. Someexamples of information query, retrieval and presentation applicationsfor the invention include, but are not limited to the following:

-   -   1. White pages and yellow pages lookups to find, email        addresses, telephone numbers, street addresses and other        information for businesses and individuals. These services can        be used in conjunction with other services, including remote        ordering and payment, offers and promotions, mapping, and        driving directions;    -   2. Management and access to personal address book, calendars and        reminders for users;    -   3. Automatic telephone dialing, reading and sending emails,        pages, instant messaging by voice, text or video and other        communications control functions;    -   4. Selection, schedules and play list management for television,        satellite broadcast, radio or other entertainment schedule. The        available information can include reviews and other information        on programming. The system may provide device control for users;    -   5. Weather information for the local area or other locations;    -   6. Stock and other investment information including, prices,        company reports, profiles, company information, business news        stories, company reports, analysis, price alerts, news alerts,        portfolio reports, portfolio plans, or other information;    -   7. Local, national and international news information including        headlines of interest by subject or location, story summaries,        full stories, audio and video retrieval and play for stories;    -   8. Sports scores, news stories, schedules, alerts, statistics,        background and history information, or other information;    -   9. The ability to subscribe interactively to multimedia        information channels, including sports, news, business,        different types of music and entertainment, applying user        specific preferences for extracting and presenting information;    -   10. Rights management for information or content that is used or        published;    -   11. Horoscopes, daily jokes and comics, crossword puzzle        retrieval and display and related entertainment or diversions;    -   12. Interactive educational programs using local and network        material, with lesson material level that is set based on user's        profile, location of the vehicle, planned route of the vehicle,        planned activities during the trip and including, interactive        multimedia lessons, religious instruction, calculator,        dictionary and spelling, geographic information, instruction for        specialized tasks planned during the trip, language training,        foreign language translation, presentation of technical manuals,        and encyclopedias and other reference material.

It will be appreciated that the foregoing statements of the features ofthe invention are not intended as exhaustive or limiting, the properscope thereof being appreciated by reference to this entire disclosureand reasonably apparent variations and extensions thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described by reference to the preferred andalternative embodiments thereof, in conjunction with the drawings inwhich:

FIG. 1 is an overall block diagram of the system according to a firstembodiment of the invention;

FIG. 2 is an overall block diagram of the system according to a secondembodiment of the invention;

FIG. 3 is an overall block diagram of a mobile device according to anembodiment of the invention;

FIG. 4 is an overall block diagram of a fixed computer according to anembodiment of the invention;

FIG. 5 is an overall diagrammatic view of the interactive naturallanguage processing system according to an embodiment of the invention;

FIG. 6 is a schematic block diagram showing the agent architectureaccording to an embodiment of the invention;

FIG. 7 illustrates a diagram for centrally maintaining context,according to one embodiment of the invention; and

FIG. 8 illustrates a system diagram for an enhanced conversational voiceuser interface according to one embodiment of the invention.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings,and describes exemplary embodiments of the invention. Other embodimentsare contemplated and modifications may be made to the exemplaryembodiments without departing from the spirit, functionality and scopeof the invention. Therefore, the following detailed descriptions are notmeant to limit the invention.

According to one embodiment of the invention, a telematic naturallanguage speech interface and non-speech interface are provided for usein mobile environments and telematic applications. The system, orportions thereof, may be used in vehicles, while on foot or at a fixedlocation such as an office or home, or at other locations. An overallblock diagram of one embodiment of the invention is illustrated in FIG.1.

A speech unit 128 and/or a keypad 14 may be integrally coupled to amobile structure 10 or may be part of mobile devices 36, fixed home oroffice computer systems 44, or other devices. Mobile devices 36 mayinclude mobile telephones, personal digital assistants, digital radios,compact disk players, navigation systems, or other mobile devices. Themobile devices 36 may be configured to integrate with set-top boxes,alarm clocks, radios, or other electronic components. The speech unit128 and/or keypad 14 may be interfaced to a Telematics Control Unit(TCU) 28 through one or more data interfaces 26. According to someembodiments, the main speech-processing unit 98 may be embedded in oneor more TCU 28. In some embodiments, the components of the speech unit128 may also be distributed between one or more TCUs.

A speech-processing unit may be built into mobile devices 36 and may becoupled with the data interfaces 26 though a wireless, or wired,handheld interface 20. Other user interface peripherals may be connectedto the TCU through the data interfaces and may include displays 18, suchas touch screen displays for displaying text, graphics and video;keypads 14 for receiving textual data input; video cameras 16 forreceiving multimedia communications or conferences; a pointing device orstylus, or other user interface peripherals. Other devices may beconnected to the TCU through the data interfaces including wide-area RFtransceivers 24, navigation system components 22, or other devices. Thenavigation system may include several components such as, GlobalPositioning System (GPS) receivers or other radiolocation systemreceivers, gyroscopes or other inertial measurement equipment, distancemeasurement sensors, such as odometers, or other components.Radiolocation equipment may receive coded signals from one or moresatellites or terrestrial sources 40. The one or more location serviceservers 48 may assist the navigation system. Other systems that canconnect to the TCU through the data interfaces may include automotivecontrol computers, digital control interfaces for devices such as mediaplayers or other electronic systems, measurement sensors, or otherspecialized electronic equipment.

The control and device interfaces 30 may connect the TCU 28 to variousdevices 32 within the mobile structure 10. The control and deviceinterfaces 30 may be used to execute local or remote commands from usersof the natural language interface. In some cases, the control and deviceinterfaces 30 may include specialized hardware for interacting withdifferent types of devices. The hardware interfaces may include analogor digital signal interfaces for device control, along with analog ordigital interfaces for measurements that may control devices 32. Theseinterfaces may also include specialized software that encapsulates orabstracts specific behavior of the devices 32. The interface softwaremay include one or more drivers that are specific to the hardwareinterface and to one or more agents. The domain agents may include thespecialized software behavior and data required to control a particulardevice or class of devices. New or updated behavior may be added to thesystem by updating data managers that are associated with specificdevices or class of devices. The devices 32 may include manual controlsor manual overrides 34. For safety reasons, the control and deviceinterfaces 30 may incorporate fail-safe systems that, for example, mayverify operating limits before changing settings, ensuring that commandsdo not conflict with settings from manual controls, and will not, insome combination with other commands or device settings, create anunsafe situation. The software behavior and data that ensures safeoperations may be included within the domain agent that is specific tothe device or class of devices. Examples of devices and system that canbe controlled through the control and device interfaces 30 include,power management systems, measurement sensors, door locks, windowcontrols, interior temperature controls, shifting of the transmission,turn signals, lights, safety equipment, engine ignition, cruise control,fuel tank switches, seat adjustments, specialize equipment such aswinches, lifting systems or loading system, and other systems.

The wide-area RF transceiver 24 may communicate with one or morewide-area wireless networks 38, which may be connected to data networks42, including the Internet, the Public Switched Telephone Network (PSTN)42 or other data networks. The wide-area wireless networks can be of anysuitable terrestrial or satellite based type. Mobile devices 36 maycommunicate with one or more local or wide-area wireless networks. Homeor office systems 44, equipped with wired or wireless networkinterfaces, may communicate through the data networks or PSTN.

According to one embodiment of the invention, data and agents may bestored and synchronized in mobile structures 10, mobile devices 36and/or fixed systems 44 having one or more main speech-processing units98. The synchronization between these different systems can occur on thewide area wireless network 38, the data network 42, through the handheldinterface 20, or other local data connections. The synchronization maybe performed automatically when any two or more of the computers areconnected to these networks. Alternatively, the synchronization may beperformed on demand under user control. The synchronization processattempts to determine which version of a data element or an agent arethe newest or most up-to-date and propagates that element. Thus,synchronization is an incremental change processes. In some cases, acomplete replacement of a database, a portion of the database or of oneor more agents may be performed rather than performing a series ofincremental updates.

The wide-area wireless networks 38, the data networks 42 or PSTN, mayconnect mobile structures 10, mobile devices 36 and fixed computers 44to one or more servers that provide one of more services. According toone embodiment of the invention, an interactive natural language userinterface may be provided that supports a transfer of data ortransmission of speech, text, video and other formats. For data centricapplications, a standardized data transfer format may be used including,for example, Hypertext Markup Language over Hypertext Transfer Protocol(HTTP), Extensible Markup Language (XML), and other data formats orschemas, over HTTP or other transfer protocol, Electronic DataInterchange formats over a variety of transport protocols, etc. It willbe understood that the exact configuration of the servers may bedetermined on a case-by-case basis with consideration being given to theexact combinations of services being offered, the service providersproviding the services, the contractual relationships between theservice provider, and other factors. The invention supports any suitableconfiguration. In each case these servers may themselves be distributedover one or more public or private networks. Some examples of serversthat may be used to deliver these services, are given below:

-   -   1. One or more payment service providers 56 that supply payment        capabilities to users. These payment services may include        electronic wallet capabilities for one or more payment accounts        and may include payment security information, payment account        information, transaction histories, account balance information,        and other information. Exemplary payment types include, stored        value accounts, promotional accounts, credit accounts,        telecommunications billing accounts, debit accounts, or other        payment types that are accessed using online or offline methods.        Payments can be computed in any manner including payment for a        specific good or service, subscription payment, metered payment,        or other payments. The payment services can be distributed in a        number of ways. Examples of computers and servers used to store        and process payment transaction information include, smart        cards, main speech processing units 128, mobile devices 36, TCUs        28, fixed personal computers 44, payment gateways, payment        servers 56, or other systems.    -   2. One or more Customer Relationship Management (CRM) systems 52        may supply any number of consumer and business customer services        as discussed above. The CRM system can supply automated services        or services that are partly or completely manual. For manual        services one or more customer service representatives may use        one or more service representative workstations 54. The CRM        system and the service representative workstation may be        connected to one or more data networks 42 or the PSTN. Any other        servers may also have connections to one or more service        representative workstations, which may be in common or        independent of each other.    -   3. One or more specialized service servers 50 may support        specialized consumer and business services.    -   4. One or more location service servers 48 may supply location        information and location based services. The location data is        used as a data input to the location services that may then be        distributed in any suitable manner including, on main speech        processing units 128, mobile devices 36, TCUs 28, fixed personal        computers 44, other servers (i.e. 46, 50, 52, 56) or the one or        more location services servers 48.    -   5. One or more emergency services servers 46 may supply both        public and private emergency services to the users.

Main speech processing unit 98, speech unit 128 and keyboard 14 may bedistributed in mobile devices 36 in a number of ways. For example, theseunits may be attached to the mobile devices 36 as independent componentsor as a single integrated component. In another embodiment of theinvention, some or all of the main speech processing unit 98 and speechunit 128 may be embedded in one or more of the TCUs 28, mobile devices36, fixed computer systems 44, or other devices.

FIG. 2 illustrates an alternative block diagram of a second embodimentof the invention. In this embodiment, the main speech processing unit 98and speech unit 128 are external to the TCU 28. These components may behoused in one or more packages or may be included in a single integratedpackage.

In all other respects, the second embodiment resembles the firstembodiment illustrated in FIG. 1. It will be understood that the exactdistribution and packaging of the main speech processing unit 98 andspeech unit 128 may be determined by the details of the deploymentsituation and should not change the functionality, capabilities orspirit of the invention in any way.

FIG. 3 illustrates a block diagram of one embodiment of the mobiledevices 36 that include main speech processing unit 98 and speech unit128 embedded in the mobile device 36, mobile telephone or other mobiledevice. The main speech processing unit may interface to one or moreprocessing units 70 located in the mobile device 36. The processingunits 70 may include one or more central processing units, one or moredata and address busses, data interfaces, volatile memory, or othercomponents. The processing units 70 may use one of more types ofnonvolatile memory 80 for software and data storage. Suitable types ofnonvolatile memory 80 may include flash memory and hard disk drives. Insome embodiments, the main speech processing unit 98 may be integratedwith the one or more processing units 70.

According to one embodiment, users may interact with the mobile device36 through the speech unit 128, the keypad 74 or keyboard, a display 72that displays text, graphics, video, or other peripheral. In someembodiments the display may be a touch screen type. Alternatively, apointing device (not shown) may be used, among other devices.

The mobile device 36 may connect to one or more wired or wirelesswide-area or local-area networks through one or more interfaces. Awide-area network transceiver 78 may connect to the wide-area wirelessnetwork 38 or the data network 42 using a wireless or wired connection,including an IP connection, a dial-up PSTN network connection or otherconnections. The local-area network transceiver 76 may connect to wiredor wireless local area networks. These networks may include the handheldinterface 20 or connections to fixed computer systems 44. In both mobiledevice environments and fixed computer environments, communicationsbetween humans and machines may not provide accurate results at leastbecause natural language commands may be incomplete in their definition.The occurrence of inaccurate results may be reduced by leveragingcontext of previous utterances, knowledge of the domain and/or theuser's history of interests and preferences or other factors.

According to one embodiment of the invention, processing may beperformed at the mobile devices 36. Alternatively, the processing may beperformed on the server side. In yet another embodiment, the processingmay occur on both the mobile devices 36 and the server side. Thecommands may be processed on-board the mobile devices 36 to enable themobile devices 36 to control themselves and/or to control other mobiledevices 36, fixed computers 44, mobile telephones, or other devices.Additionally, mobile devices 36 may track context. According to anotherembodiment of the invention illustrated in FIG. 7, a context manager 702may be centrally maintained to allow input to and output from multiplemobile devices 36. Each of the mobile devices may communicate with thecontext manager 702 to register via registration module 712, whereinregistration may indicate events that the mobile devices 36 maysubscribe to. The context manager 702 may receive input in Context XMLform, for example. The other registered mobile devices 36 may beinformed of context changes through context tracking module 714, therebysynchronizing the context across the registered mobile devices 36.According to one embodiment of the invention, registered mobile devices36 may be added or removed. The registered mobile devices 36 may bedynamic link libraries (DLLs), object code, or other data that may bespecific to the mobile device 36.

According to yet another alternative embodiment of the invention,context information may be determined from a command or request that ispresented as a text message and/or a command or request that ispresented as a verbal utterance and processed using a multi-passautomatic speech recognition module that transcribes the verbalutterance to a text message. The multi-pass automatic speech recognitionmodule may use a dictation grammar or a large vocabulary grammar, amongother resources, to transcribe the verbal utterance into a text message.On platforms where a dictation grammar is not available, the multi-passautomatic speech recognition may use a virtual dictation grammar thatuses decoy words for out-of-vocabulary words. Decoy words may includeutility words, nonsense words, isolated syllables, isolated distinctsounds and other decoy words.

According to one embodiment of the invention, the text message may besearched for a particular character, group of characters, word, group ofwords, and other text combinations. The text combination may be comparedagainst entries in a context description grammar that is associated witheach agent 106. If a match is identified between an active grammar inthe context description grammar and the command and/or request, then thematch may be scored. The agents 106 may be ranked based on thedetermined score. In generating an aggregate response from the one ormore responses received from the agents 106, the ordering of theresponses from the individual agents may be determined based on the rankof agents 106. The aggregate response may be generated by a responsegenerator module. Agents 106 may update a context stack, that includesan ordered list of command contexts, to enable follow-up requests.

According to another embodiment of the invention, if a match is notfound, or only a partial match is found, between the text message andactive grammars, then a knowledge-enhanced speech recognition system maybe used to semantically broaden the search. The knowledge-enhancedspeech recognition system may be used to determine the intent of therequest and/or to correct false recognitions. The knowledge-enhancedspeech recognition may access a set of expected contexts that are storedin a context stack to determine a most likely context. Theknowledge-enhanced speech recognition may use context specific matchersthat are able to identify context such as time, location, numbers,dates, categories (e.g., music, movies, television, addresses, etc.) andother context. The matching may be performed by comparing a character,group of characters, a word, group of words, and other textcombinations. Alternatively, or in addition to text based matching, thematching may be performed using phonetic matching, among othertechniques. The results of any match may be used to generate a commandand/or request that is communicated to agents 106 for additionalprocessing. According to one embodiment of the invention, non-speechinterface 114 may show system, state and history information in a moreconcise manner than is possible through the speech interface. Non-speechinterface 114 may be accessed to create or extend capabilities of agents106. These operations may include scripting of agents, adding data tothe agent or databases 102 used by the agent, adding links toinformation sources, among other operations.

According to another embodiment of the invention, mobile devices 36 maybe configured to allow speech annotations of objects stored thereon. Theobjects may include photographs, calendar entries, email messages,instant messages, phonebook entries, voice mail entries, digital moviesor other objects. Mobile devices 36 may transcribe the speechannotations to textual annotations and store the textual annotationswith the object. Alternatively, mobile devices 36 may be configured toenable users to input non-speech annotations, such as textualdescriptions, that are stored along with the objects.

According to one embodiment of the invention, the annotated objects maybe stored on a server side, a client side, a combination of server sideand client side, or according to other configurations. The inventionfurther contemplates collaboratively exchanging and sharing theannotated objects among distributed workgroups that may includecentralized servers having shared workspaces for providing common objectstorage and retrieval facilities. The shared workspaces may beimplemented on the centralized servers and may be accessed fromdifferent platforms using the mobile devices 36. The system may includea peer-to-peer system for accessing the annotated objects.

According to one embodiment of the invention, the non-speech annotationsmay be classified and searched. In an alternative embodiment, the speechannotations may be classified and searched. However, classifying andsearching speech annotations may be considerably more difficult thanclassifying and searching non-speech annotations.

According to another embodiment of the invention, the textualannotations and/or textual descriptions that are associated with theobjects may be stored as metadata, thereby enabling searching for theobjects using the metadata. The metadata may include GPS information,environmental information, geographic information, or other information.For example, proximity to famous landmarks may be determined using GPSinformation, environmental information, geographic information, or otherinformation, and this information may be integrated into the metadataassociated with the objects. According to one embodiment of theinvention, GPS coordinates may be stored in the metadata associated withthe objects and users may search for selected objects based on the GPScoordinates. A user may provide a-speech command such as “show me allthe photos of Greece.” In this case, the system would limit the type ofobject to photographs and would determine the GPS coordinates of Greece.The system would then search the metadata for objects that correspond tophotographs and that also satisfy GPS coordinates for Greece. Accordingto another embodiment of the invention, including GPS coordinates in themetadata of objects enables post-processing of the objects based on theGPS coordinates. For example, the objects may be subjected to an initialsort based on generalized GPS coordinates that are stored in themetadata and may be subjected to additional sorting based on moreparticular criteria for the GPS coordinates. Thus, a user may initiallysearch for object metadata that corresponds to a location near a famouslandmark and may use image matching to label the objects with thesearchable metadata (i.e., textual descriptions). For example, using theGPS coordinates stored in the metadata, users may first determine thatphotographs were taken at the Jefferson Memorial and may use thisinformation to label the photographs with searchable metadata (i.e.,textual descriptions) including “photo of the Jefferson Memorial.” Oneof ordinary skill in the art will readily appreciate that metadata mayinclude various types of information and may be searched using thevarious types of information.

According to one embodiment of the invention, the textual annotationsmay be communicated using a short message service on the mobiletelephone or other device. Short message service is a text messageservice that enables sending and receiving of short textual messages.The textual messages may be stored at data centers for forwarding tointended recipients. Other configurations may be used.

According to another embodiment of the invention, the mobile devices 36may support multi-modal communications that enable displaying ofnon-speech search results on a graphical interface and receipt of speechcommands to provide a follow-up search, among other configurations. Forexample, the user may be presented with textual search resultscorresponding to a name of a famous person and the user may provide aspeech command to find a biography of the famous person. The system maymaintain the context of the textual search results to find the biographyassociated with the famous person. By contrast, known systems mayperform a follow-up search on the term “biography” and may present adictionary definition of the term “biography.”

FIG. 4 illustrates an embodiment that uses a fixed computer 44. In someembodiments, the main speech processing unit 98 and speech unit 128 maybe embedded into the fixed computer 44. The main speech processing unitmay interface with one or more processing units 84 that are associatedwith the fixed computer. The processing units may include one or morecentral processing units, one or more data and address busses, datainterfaces, volatile memory, or other components. The processing unitmay use one of more types of nonvolatile memory 94 for software and datastorage. Suitable types of nonvolatile memory include, for example,flash memory and hard disk drives. In some embodiments, the main speechprocessing unit 98 may be integrated with the one or more processingunits 84. According to one embodiment of the invention, processing maybe performed at the fixed computers 44. The commands may be processedon-board to enable the fixed computers 44 to control themselves and/orto control other fixed computers 44, mobile devices 36, mobiletelephones, or other devices. Additionally, fixed computers 44 may trackcontext as described above.

According to another embodiment of the invention, fixed computers 44 maybe configured to allow verbal annotations of objects stored thereon.Fixed computers 44 may transcribe the verbal annotation to text andstore the textual annotation with the object. Alternatively, fixedcomputers 44 may be configured to enable users to manually input textualdescriptions that are stored along with the objects. According to oneembodiment of the invention, the textual annotations and/or textualdescriptions may be classified and searched. In an alternativeembodiment, fixed computers 44 may classify and search the verbalannotations rather than textual annotations. However, classifying andsearching verbal annotations may be considerably more difficult thanclassifying and searching textual annotations and/or textualdescriptions.

In another embodiment of the invention, users may interact with fixedcomputers 44 using speech unit 128, the keyboard 88 or keypad, a display86 for displaying text, graphics, video, or other peripherals. Accordingto some embodiments of the invention, the display may be a touch screentype. Alternatively, a pointing device (not shown) may be used, alongwith other devices. Fixed computers 44 may be coupled to one or morewired or wireless wide-area or local-area networks through one or moreinterfaces. A wide-area transceiver 92 may connect to the wide-areawireless network 38 or the data network 42, using a wireless or wiredconnection, including an IP network, a dial-up PSTN network connection,or other connections. The local-area network transceiver 90 may connectto wired or wireless local area networks. These networks may includeconnections to mobile devices 36.

In order for devices to properly respond to requests and/or commandsthat are submitted in a natural language form, machine processablerequests and/or algorithms may be formulated after the natural formquestions or commands have been parsed and interpreted. Algorithmsdescribe how the machines should gather data to respond to the questionsor commands. Depending on the nature of the requests or commands, theremay not be a simple set of requests and/or algorithms that will returnan adequate response. Several requests and algorithms may need to beinitiated and even these requests and algorithms may need to be chainedor concatenated to achieve a complete response. Further, no singleavailable source may contain the entire set of results needed togenerate a complete response. Thus, multiple requests and/or algorithms,perhaps with several parts, may be generated to access multiple datasources that are located both locally or remotely. Not all of the datasources, requests and/or algorithms may return useful results or anyresults at all. Useful results that are returned are often embedded inother information and may need to be extracted from the otherinformation. For example, a few key words or numbers may need to be“scraped” from a larger amount of other information in a text string,table, list, page, video stream or other information. At the same time,extraneous information including graphics or pictures may be removed toprocess the response. In any case, the multiple results must beevaluated and combined to form the best possible response, even in caseswhere some requests do not return useful results or fail to produceresults entirely. In cases where the command is determined to beambiguous or the result is inherently subjective, determining theresults to present in the response is a complex process. Finally, tomaintain a natural interaction, responses should be returned to the userrapidly. Managing and evaluating complex and uncertain requests, whilemaintaining real-time performance, is a significant challenge.

The invention provides a complete speech-based command generation,information query, retrieval, processing and presentation environment ora combination of speech-based and non-speech-based command generation,information query, retrieval, processing and presentation environmentfor telematic applications. In addition, the invention may be useful forcontrolling the system itself and/or external devices. This integratedenvironment makes maximum use of context, prior information and domainand user specific profile data to achieve a natural environment for oneor more users submitting requests or commands in multiple domains.Through this integrated approach, a complete speech-based naturallanguage command, algorithm and response environment or a combination ofspeech-based and non-speech-based command, algorithm and responseenvironment may be created.

The telematic natural language interface may be deployed as part of, ora peripheral to a TCU or other mobile devices 36, as part of a mobiledevice interfaced to vehicle computers and other mobile system throughwired, wireless, optical, or other types of connections or fixedcomputers interfaced to the vehicle computers or other systems through acombination of wired, wireless, optical and/or other types ofconnections. Alternatively, the components of the interactive naturallanguage telematic interface may be distributed in any suitable mannerbetween these multiple computing platforms.

FIG. 5 illustrates one exemplary schematic diagram view of a system 90according to an embodiment of the invention. The system 90 may include amain unit 98, speech units 128, and multi-modal devices 155. In analternative embodiment, the system 98 may include completely separatesystems for main unit 98, speech units 128 and multi-modal devices 155.The event manager 100 may mediate interactions between components ofmain unit 98. The event manager 100 provides a multi-threadedenvironment allowing the system 98 to operate on multiple commands orquestions from multiple user sessions, without conflict, and in anefficient manner to maintain real-time response capabilities.

FIG. 8 illustrates one exemplary embodiment of an enhancedconversational user interface 800 that receives multi-modal input fromusers 802 and communicates with a conversational speech analyzer 804.The conversational speech analyzer 804 may be coupled to severalcomponents, including a general cognitive model 806, an environmentalmodel 808, a personalized cognitive model 810, and an adaptivemisrecognition analysis engine 812 or other components. According to oneembodiment of the invention, the conversational speech analyzer 804 mayinclude one or more speech recognition engines that transcribeutterances to textual messages. The transcription may be performed usingone or more conversational grammars, a context description grammar 112,or other transcription methods. According to one embodiment of theinvention, data that is entered in a text format may be merged with datathat is transcribed to a textual format from the utterance.Conversational speech analyzer 804 also may include a semanticknowledge-based model that analyzes the textual message and detectscommand components.

According to one embodiment of the invention, personalized cognitivemodel 810 is a model derived from a user's interaction pattern with thesystem and may be used to predict what actions the user may take next intime, thus assisting with speech recognition and/or question or commandrecognition. Personalized cognitive model 810 may track actionsperformed by the user. When the system is attempting to predict userbehavior, the personalized cognitive model may be consulted first. Thesystem may have multiple personalized cognitive models, wherein one maycorrespond to each user.

According to another embodiment of the invention, general cognitivemodel 806 is a statistical abstract that corresponds to interactionpatterns with the system for multiple users. Data stored within generalcognitive model 806 may be used to predict a user's next action, thusassisting with speech recognition and/or question or commandrecognition. The general cognitive model 806 may also track what actionsa particular user has performed and may be used when the user interactswith the system in a way that is not handled in the personalizedcognitive model.

According to one embodiment of the invention, the environmental model808 may include information associated with the user's environment andsurroundings. The information may include the type of environment that auser is in (e.g., quiet or noisy); details of a microphone and/orspeaker system; the user's current global position and movement, as maybe determined by GPS; current system status, such as what song/movie isplaying, is the system in the midst of retrieving something, or othersystem status; details on all voice-enabled devices in the immediatevicinity, such as a presence of a voice-enabled TV, stereo, and DVDplayer in the same room; a user's credit card information, such asnumbers and current balances, wherein the user may ask a mobiletelephone to download and pay for a video and the system may respondthat there are insufficient finds; or other information. The informationmay be accessed to invoke a context, domain knowledge, preferences,and/or other cognitive qualities that enhance the interpretation ofquestions and/or commands.

The conversational speech analyzer 804 may also access the generalcognitive model 806 and/or the personalized cognitive model 810 tofurther refine a context, domain knowledge, preferences, and/or othercognitive qualities to enhance the interpretation of questions and/orcommands. Based on information received from general cognitive model806, environmental model 808 and/or the personalized cognitive model810, the system may enhance responses to commands and questions byincluding a prediction of user behavior.

Adaptive Misrecognition Analysis Engine 812 may analyze and storetextual messages, including transcribed utterances, that are identifiedas being unrecognized or incorrectly recognized by conversational speechanalyzer 804. Upon a determination that text is unrecognized, the systemmay generate an unrecognized event. For example, an unrecognized eventmay result from not finding a match to text and/or the transcribedutterance.

According to one embodiment of the invention, the system may implementone or more techniques to determine that textual messages areincorrectly recognized. For example, the user may command the system toplay a specific song title and the system may misrecognize the requestedsong title and provide a song having a different title, the system maymisrecognize an incorrect song title and provide a song having adifferent than requested title, the system may misrecognize a correctsong title and provide a song having an invalid song title, among othervariations. When the system misrecognizes a request, the user typicallyprovides immediate feedback such as, overriding the command in a timeshorter than the expected execution time of the command, repeating theoriginal request, issuing a stop command, or taking other action,wherein the action may be presented verbally, non-verbally by pushing aselected button on a cell phone or remote control, or both, among otherconfigurations. According to one embodiment of the invention, the systemmay detect the user action and may prompt the user to re-phrase therequest to enable the system to modify words in the query. The user'sactions may be analyzed in a statistical model to determine a frequencyoccurrence of misrecognitions for particular commands, with the resultsbeing used to update the corresponding personalized cognitive model 810.

According to another embodiment of the invention, the conversationalspeech analyzer 804 may access the personalized cognitive model 810 toproactively select a next best (or nth best) match for the receivedtext. A match may be confirmed by user action that include the user notimmediately canceling the command or taking other action. Themisrecognitions may also be analyzed to potentially determinepersonalized tuning parameters for the speech recognition components ofthe system. For example, the system may, over time, tune the speechrecognition engine to increase recognition by analyzing how the speechrecognition engine misrecognizes the utterance.

The event manager 100 may mediate interactions between other componentsof the invention. The event manager can provide a multi-threadedenvironment allowing the system to operate on multiple commands orquestions from multiple user sessions without conflict and in anefficient manner, maintaining real-time response capabilities.

Agents 106 may include a collection of grammars, criteria handlers, andalgorithms that are accessible to respond to a set of requests and/orcommands. Agents 106 may further contain packages of both generic anddomain specific behavior for the system 98. Agents 106 may usenonvolatile storage for data, parameters, history information, andlocally stored content provided in the system databases 102 or otherlocal sources. One or more user profiles 110 may be provided thatinclude user specific data, parameters, and session and historyinformation for determining the behavior of agents 106. One or morepersonality modules 108 may be provided in a data determining systemthat include personality characteristics for agents. The update manager104 manages the automatic and manual loading and updating of agents 106and their associated data from the Internet 146 or other network throughthe network interface 116.

According to one embodiment of the invention, the speech-based interfacefor the system 90 may include one or more speech units 128. Speech units128 may include one or more microphones, for example array microphone134, to receive the utterances from the user. The speech received at themicrophone 134 may be processed by filter 132 and passed to the speechcoder 138 for encoding and compression. In one embodiment, a transceivermodule 130 may transmit the coded speech to the main unit 98. Codedspeech received from the main unit 98 is detected by the transceiver130, then decoded and decompressed by the speech coder 138 andannunciated by the speaker 136.

According to one embodiment of the invention, the non-speech-basedinterface for the system 90 may include one or more multi-modal devices155 that may include mobile devices, stand alone or networked computers,personal digital assistances (PDAs), portable computer devices, or othermulti-modal devices.

The speech units 128, multi-modal devices 155 and the main unit 98 maycommunicate over a communication link. The communication link mayinclude a wired or wireless link. According to one embodiment, thecommunication link may comprise an RF link. The transceiver 130 on thespeech unit may communicate coded speech data bi-directionally over thecommunication link with the transceiver 126 on the main unit 98.According to another embodiment, the RF link may use any standard localarea wireless data protocols including the IEEE 802.11, Bluetooth orother standards. Alternatively, an infrared data link conforming to anysuitable standard such as IrDA or other infrared standards can be used.In an alternative embodiment, wires may connect the speech unit 128 andthe main unit 98, eliminating the need for one speech coder 138. Otherwired or wireless analog or digital transmission techniques can be used.

According to one embodiment of the invention, coded speech received atthe transceiver 126 on the main unit 98 is passed to the speech coder122 for decoding and decompression. The decoded speech may be processedby the speech recognition engine 120 using the context descriptiongrammar module 112, among other information. Any recognized informationmay be processed by the parser 118, which transforms information intocomplete algorithms and questions using data supplied by knowledgeagents. Knowledge agents may be selected from the plurality of agents106 using a grammar stack, wherein the knowledge agents provideinformation for generating a response to the question or command. Theknowledge agents may then process the commands or questions by creatingrequests that are submitted to local databases 102 or submitted throughthe network interface 116 to external data sources over the Internet 146or external other networks. Algorithms typically result in actions takenby the system 90 itself (i.e., pause or stop), or to a remote device ordata source (i.e., download data or program, or control a remotedevice), through the network interface to the Internet or other datainterface.

According to one embodiment of the invention, knowledge agents mayreturn results of questions as responses to users. The responses may becreated using the results of information requests, the systempersonality 108, the user preferences, other data in the user profile110, and/or other information. Agents 106 may present these resultsusing the speech unit 128. The agents 106 may create a response string,which is sent to the text to speech engine 124. The text to speechengine 124 may generate the utterances, which may be encoded andcompressed by the speech coder 122. Once coded, the utterances aretransmitted from the main unit 98 by the transceiver 126 to thetransceiver 130 on the speech unit 128. The utterance is then decodedand decompressed by the speech coder 138 and output by the speaker 136.Alternatively, agents 106 may present the results using multi-modaldevices 155.

The non-speech interface 114 may be part of, or separate from, themulti-modal devices 155 and may be used as a substitute or complement tothe speech interface. For example, non-speech interface 114 may be usedto present and interact with non-speech (e.g., graphical or tabular)information in a manner more easily understood by the user. According toone embodiment of the invention, multimodal support may be provided tomaintain the context during both voice interaction and interactionthrough the non-speech interface 114. In one exemplary embodiment, acommunication channel may be opened between multimodal devices 155 andthe main user interface system 90 to allow multimodal devices 155 toinput text commands and questions. According to another embodiment ofthe invention, multimodal devices 155 may send a string of text orkeywords for a command or question. The main interface system 90 maysynchronize the context between multimodal device 155 and the speechunits 128. In order to send a response to the corresponding device, themain user interface system 90 may track where the input came from sothat the response may be sent to a TTS or multi-modal device 155.

According to one embodiment of the invention, non-speech interface 114may show system, state and history information in a more concise mannerthan is possible through the speech interface. Non-speech interface 114may be accessed to create or extend capabilities of agents 106. Theseoperations may include scripting of agents, adding data to the agent ordatabases 102 used by the agent, adding links to information sources,among other operations.

According to another embodiment of the invention, system 90 may includedifferent types of agents 106. In some embodiments of the invention,generic and domain specific behavior and information may be organizedinto domain agents 156. The system agent, on the other hand, may providedefault functionality and basic services. The domain specific agents mayprovide complete, convenient and re-distributable packages or modulesfor each application area. In other words, the domain agents may includedata that is needed to extend or modify the functionality of the system90 in a current or new domain. Further, domain agents and theirassociated data can be updated remotely over a network as new behavioris added or new information becomes available. Domain agents may accessa plurality of sources that may provide various services. Domain agentsmay use the services of other, typically more specialized, data managersand system agent. Agents are distributed and redistributed in a numberof ways including on removable storage media, transfer over networks orattached to emails and other messages. The invention may provide licensemanagement capability allowing the sale of data managers by thirdparties to one or more users on a one time or subscription basis. Inaddition, users with particular expertise can create data managers,update existing data managers by adding new behaviors and informationand making these data managers available to other users. A block diagramof the agent architecture according to an embodiment of the invention isshown in FIG. 6.

Agents 106 may receive and return events to the event manager 100. Bothsystem agents 150 and domain agents 156 may receive questions andcommands from the parser 118. Based on keywords in the questions andcommands and the structures of the questions and commands, the parsermay invoke selected agents. Agents use the nonvolatile storage for data,parameters, history information and local content provided in the systemdatabases 102.

According to one embodiment of the invention, when the system starts-upor boots-up, the agent manager 154 may load and initialize the systemagent 150 and the one or more domain agents 156. Agent manager 154includes knowledge of agents 106 and maps agents 106 to the agentlibrary 158. At shutdown, the agent manager may unload the agents 106.The agent manager 154 also performs license management functions for thedomain agents 156 and content in the databases 102.

The system agent 150 manages the criteria handlers 152 that handlespecific parameters or values (criteria) used to determine context forquestions and commands. According to one embodiment of the invention,criteria handlers 152 include parsing routines that are specialized torecognize particular parts of speech, such as times, locations, movietitles, and other parts of speech. The criteria handlers 152 mayidentify matching phrases and extract semantic attributes from thephrases. Both the system agent 150 and the domain agents 156 may use thecriteria handlers 152. The various domain agents 156 may use theservices of the system agent 150 and of other, typically morespecialized, domain agents 156. The system agent 150 and the domainagents 156 may use the services of the agent library 158, which containsutilities for commonly used functions. According to one embodiment ofthe invention, the agent library may be a dynamic link library thatimplements one or more agents. The agent library may include utilitiesfor text and string handling, network communications, database lookupand management, fuzzy and probabilistic evaluation, text to speechformats, and other utilities.

Domain agents 156 may be data-driven, scripted or created with compiledcode. A base of generic agent may be used as the starting point fordata-driven or scripted agents. Agents created with compiled code aretypically built into dynamically linkable or loadable libraries.Developers of agents can add new functionality to the agent library 158as required. Details of agent distribution and update, and agentcreation or modification are discussed in sections below.

According to another embodiment of the invention, capabilities areprovided to distribute and update system agents 150, domain agents 156,agent library 158 components, databases 102, and context descriptiongrammar 112 over wireless or wired networks 136, including IP networksand dial-up networks using the update manager 104. The network interface116 may provide connections to one or more networks. The update manager104 may also manage the downloading and installation of core systemupdates. The agent manager 154 may perform license management functionsfor the domain agents and the databases. The update manager 104 andagent manager 154 may perform these functions for all agents anddatabase content including, agents and content available to all users oragents and/or content available to selected users. Examples of agent anddatabase components added or updated on a periodic basis include:

-   -   agents for new domains;    -   additional domain knowledge for agents;    -   new keywords for a domain, which can include names of        politicians, athletes, entertainers, names of new movies or        songs, etc. who have achieved recent prominence;    -   links to a preferred set of information sources for the domains        covered including links for, entertainment, news, sports,        weather, etc.;    -   updates to domain information based on, for example, changes to        tax laws, company mergers, changing political boundaries;    -   updates to content, including dictionaries, encyclopedias and        almanacs; and    -   other content and database components.

When a user requires or selects a new domain agent 156 or databaseelement 102, the update manager 104 may connect to their source on thenetwork 146 though the network interface 116, download and install theagent or data. To save system resources and to comply with any licenseconditions, the update manger 104 may uninstall agents that are nolonger in use. In one embodiment of the invention, the update manager104 may periodically query one or more sources of the licensed agentsand database components to locate and download updates to agentexecutables, scripts or data as they become available. Alternatively,the agent sources may initiate the downloading of agent updates of theregistered or licensed agents to the update manager as they becomeavailable.

The agent manager 154 may provide license management clients that arecapable of executing most any license terms and conditions. When aparticular agent 106 and/or database element 102 is selected based on asubmitted command, the agent manager 154 verifies that the use of theagent or data element is within the allowed terms and conditions, and ifso, invokes the agent or allows access to the data element. Licensemanagement schemes that can be implemented through the agent manager 154include outright purchase, subscription for updates, one time or limitedtime use. Use of shared agents and data elements (such as thosedown-loaded from web sites maintained by groups of domain experts) mayalso be managed by the agent manager 154.

If questions or commands do not match an agent that is currently loadedon the system, the agent manager 154 may search the network 146 throughthe network interface 116 to find a source for a suitable agent. Thisprocess may be triggered, for example, when a query is made in a domainfor which an agent is not available, when a new device is added to amobile structure, or when the behavior of a mobile device is updated.Once located, the agent can be loaded under the control of the updatemanager 104, within the terms and conditions of the license agreement,as enforced by the agent manager.

New commands, keywords, information, or information sources can be addedto any domain agent 156 by changing agent data or scripting. Theseconfiguration capabilities may allow users and content developers toextend and modify the behavior of existing domain agents 156 or tocreate new domain agents 156 from a generic agent without the need tocreate new compiled code. Thus, the modification of the domain agents156 may range from minor data-driven updates by even the most casualusers, such as specifying the spelling of words, to development ofcomplex behavior using the scripting language as would typically be doneby a domain expert. The user can create and manage modifications todomain agents 156 through speech interface commands or using non-speechinterface 114. User-specific modifications of domain agents 156 arestored in conjunction with the user's profile 110 and accessed by thedomain agent 156 at run-time.

The data used to configure data driven agents 156 may be structured in amanner to facilitate efficient evaluation and to help developers withorganization. This data is used not only by the agent, but also in thespeech recognition engine 120, the text to speech engine 124, and theparser 118. Examples of some major categories of data include:

-   -   1. Content packages may include questions or commands. Each        command or question or group of commands or questions may        include contexts used for creation of one or more requests. The        domain agent 156 may pass a context description grammar        expression to the parser 118 for evaluation of a context or        question. An initial or default context is typically supplied        for each command or question. The command or question includes a        grammar for the management and evaluation of the context stack.    -   2. Parameters and other operating data on devices that are under        control of the natural language speech interface. The domain        agent 156 may use this data and parameters to determine how to        execute a command, how to formulate the command string for the        parser 118, determine if the command is feasible, and determine        if the command can be executed within safety and operating        limits.    -   3. Page lists or pointers to other local or network content        sources. For each page or content source, there may be a pointer        (e.g. URL, URI, or other pointer) to the page or source. Each        page may have specific scraping information used to extract the        data of interest. The scraping information may include, for        example, matching patterns, HTML or other format parsing        information.    -   4. A response list, determining the response of the domain agent        156 to a particular command or question given the context, the        user profile and the information retrieved. Responses may        include diagnostic error messages or requests for more        information if the question or command cannot yet be resolved        from the known information. Responses can be based on or be        dependent on thresholds or probabilistic or fuzzy weights for        the variables.    -   5. Substitution lists that include variable substitutions and        transformations, often applied by the agents 150, 156 in the        formatting of requests and results. For example, a stock domain        specific agent 156 would use a substitution list of company        trading symbols, company names and commonly used abbreviations.        Substitutions and transformations can be performed on commands        and questions to create precise queries, which can be applied        against one or more information sources or to results for        creating more meaningful output to the user. Substitution lists        also include information for optimally dealing with structured        information, such as HTML formatted page parsing and evaluation.    -   6. Personalities used for responses. Personalities can be        constructed by combining multiple traits in a weighted manner.        Weights can be specified for each agent's domain area to create        one or more specific personalities. Examples of personality        traits include, sarcasm, humor, irritation, and sympathy, and        other traits.    -   7. Public and user specific parameters for sources,        substitutions, transformations, variables or criteria. The        public parameter lists are part of the agent package 156. The        user specific parameters are included in the user profile 110.

Commands and questions are interpreted, requests formulated, responsescreated and results presented can be based on the user's personal oruser profile 110 values. Personal profiles may include informationspecific to the individual, their interests, their special use ofterminology, the history of their interactions with the system, domainsof interest, or other factors. The personal profile data can be used bythe agents 106, the speech recognition engine 120, the text to speechengine 124, and the parser 118. Preferences can include, special(modified) commands, past behavior or history, questions, informationsources, formats, reports, alerts or other preferences. User profiledata can be manually entered by the user and/or can be learned by thesystem based on user behavior. User profile values can include: 1)spelling preferences; 2) date of birth for user, family and friends; 3)income level; 4) gender; 5) occupation; 6)location information such as,home address, neighborhood, and business address, paths traveled,locations visited; 7) vehicle type or types; 8) vehicle operatorcertifications, permits or special certificates; 9) history of commandsand queries; 10) telecommunications and other service providers andservices; 11) financial and investment information; 12) synonyms (i.e.,a nick name for someone, different terms for the same item); 13) specialspelling; 14) keywords; 15) transformation or substitution variables;16) domains of interest; or, 17) other values.

End users may use the data driven agent 156 extension and modificationfacilities and values stored in user profiles 110 to create specialreports, packages of queries, alerts and output formats. A single alertor report can be configured to use multiple data sources and othervariables (i.e., time, location, measured value) value to determine whenalerts should be sent. For example, an alert can be generated bysampling a stock price every 15 minutes and sending an alert if theprice drops below some value. In another example, an alert may begenerated when a particular condition or combination of conditions occurfor the vehicles. Alerts and reports can be directed to a local orremote output.

To create a report, the user may first specify a set of commands orquestions. Next, the user can create or select a format for the report.Finally the user may name the report. A report can have variableparameters. For example, a user may create a company stock report andexecute the report by stating its name and the company name, which givesthe user selected information in a specified format for that company. Inanother example, a user can create a “morning” report, which presentsselected multimedia information from different sources (news, sports,traffic, weather) in the order and formats desired. In yet anotherexample, the user can create a report on the status of one or morevehicle systems. Alerts and reports can be created using only voicecommands and responses, commands and responses through the graphicaluser interface 114, or a combination of the two. Reports can be runlocally or remotely with respect to the vehicle. To create a report,alert, or other specialized behavior, the user performs a number ofsteps including: 1) specifying the command to run a report or alert; 2)specifying the question or questions, including keywords, used for aquery; 3) setting the criteria for running the report such as on commandor when a particular condition is met; 4) defining preferred informationsources; 5) defining preferences for order of result evaluation bysource, value, and other parameters; 6) specifying the presentationmedium for a report or alert, such as an email, the text to speechengine, a message to a pager, or a text and graphics display; and, 7)specifying the preferred format for the report, such as information tobe presented, order of information to be presented, preferredabbreviations or other variable substitutions.

Filtering and noise elimination may be important in facilitating thevarious functionalities of system 90 and may improve operation in noisymobile environments. Recognition and parsing of the user's speech isimplemented with good signal to noise ratio at the input to the speechrecognition engine 120. To provide acceptable results, a set of acousticmodels, an array microphone 134, a filter 132, or other components, maybe employed. If a good signal to noise ratio cannot be achieved, a noiseidentification algorithm may be used and the appropriate acoustic model,for example, one that has been trained in conditions similar to theidentified noise, may be selected. According to one embodiment of theinvention, the microphone array, filters and speech coder 138 arephysically separated from the main unit 98 into a speech unit 128, andconnected using a wireless link. Since bandwidth on a wirelessconnection is at a premium, the speech coder dynamically adapts thedigitization rate and compression of the captured speech.

Some embodiments of the invention may use one or more arrays ofmicrophones 134 to provide better directional signal capture and noiseelimination than can be achieved with a single microphone. Themicrophone array can be one-dimensional (a linear array) ortwo-dimensional (a circle, square, triangle or other suitable shape).The beam pattern of the array can be fixed or made adaptive though useof analog or digital phase shifting circuitry. The pattern of the activearray is steered to point in the direction of the one or more usersspeaking. At the same time, nulls can be added to the pattern to notchout point or limited area noise sources. The use of the array microphonealso helps reduce the cross talk between output from the text to speechengine 124 through the speaker 136 or from another user talking anddetection of the user's speech.

The invention may use an analog or digital filter 132 between the arraymicrophone or conventional microphone 134 and the speech coder 138. Thepass band of the filter can be set to optimize the signal to noise ratioat the input to the speech recognition engine 120. In some embodiments,the filter is adaptive, using band shaping combined with notch filteringto reject narrow-band noise. One embodiment employs adaptive echocancellation in the filter. The echo cancellation helps prevent crosstalk between output from the text to speech engine and detection of theuser's speech as well as suppression of environmentally caused echoes.Algorithms comparing the background noise to the signal received fromthe users speech may be used to optimize the band-shaping parameters ofthe adaptive filter.

The speech received by the array microphone 134 and passed through thefilter 132 may be sent to the speech digitizer or coder 138. The speechcoder may use adaptive lossy audio compression to optimize bandwidthrequirements for the transmission of the coded speech to the speechrecognition engine 120 over a wireless link. The lossy coding isoptimized to preserve only the components of the speech signal requiredfor optimal recognition. Further, the lossy compression algorithms thatmay be used are designed to prevent even momentary gaps in the signalstream, which can cause errors in the speech recognition engine. Thedigitized speech may be buffered in the coder and the coder may adaptthe output data rate to optimize the use of the available bandwidth. Theuse of the adaptive speech coder is particularly advantageous when aband-limited wireless link is used between the coder and the speechrecognition engine.

The microphone can be complemented with an analog or digital (i.e.,Voice over IP) speech interface. This interface allows a remote user toconnect to the system and interact with it in the same manner possibleif they were physically present.

In an alternative embodiment, the array microphone can be replaced by aset of physically distributed microphones or headsets worn by the users.The distributed microphones can be placed in different parts of avehicle, different parts of a room or in different rooms of a building.The distributed microphones can create a three-dimensional array toimprove signal to noise ratio. The headset can use a wireless or wiredconnection.

While the invention is intended to be able to accept most any naturallanguage question or command, ambiguity may be a problem. To assistusers formulate concise questions and commands, the system can support avoice query language. The language may be structured to allow a varietyof queries and commands with minimal ambiguity. Thus, the voice querylanguage helps users clearly specify the keywords or contexts of thequestion or command along with the parameters or criteria. The languagecan provide a grammar to clearly specify the keyword used to determinethe context and present a set of one or criteria or parameters. A userasking a question or stating a command in the voice query language maynearly always be guaranteed to receive a response.

The voice query language may be sensitive to the contents of the contextstack, wherein a context defines a set of questions that can beactivated or deactivated during a conversation. According to oneembodiment, each agent may designate one context to be the root contextthat defines base algorithms that the agent implements. Thus, follow-onquestions can be asked using an abbreviated grammar, since key words andcriteria can be inherited from the context stack. For example, the usercan simply ask about another keyword if the criteria of the questionremain constant.

The system 90, may provide built in training capabilities to help userslearn the best methods to formulate their questions and commands. Theinteractive training allows users to audibly or visibly see the machineinterpretation of their queries and provides suggestions on how tobetter structure a query. Using the interactive training, users canquickly become comfortable with the voice query language and, at thesame time, learn how to optimize the amount of information required witheach step of a dialog.

The output of the speech coder 122 may be fed to the speech recognitionengine 120. The speech recognition engine 120 recognizes words andphrases, using information in the context description grammar 112 andpasses these to the parser 118 for interpretation. The speechrecognition engine 120 may determine the user's identity by voice andname for each utterance. Recognized words and phrases may be tagged withthis identity in all further processing. Thus, as multiple users engagein overlapping sessions, the tags added by the speech recognition engine120 to each utterance allows other components of the system 90 to tiethat utterance to the correct user and dialog. The user recognitioncapability may further be used as a security measure for applications,such as auctions or online shopping, where this is required. Voicecharacteristics of each user may be contained in the user profile 110.

According to one embodiment of the invention, users may start a dialogwith the system 90 when they first address it. This can be done byspeaking a generic word (“computer”) or addressing a specific name(“Fred”), which may be generally tied to a system personality 108. Oncethe user starts the dialog, it may be recognized by the speechrecognition engine 120, using unique characteristics of the user'sspeech. At the end of a dialog or to interrupt a dialog, the user mayutter a dismissal word (“good bye”).

According to another embodiment of the invention, the system 90 mayemploy a speech recognition engine 124 that gains improved wordrecognition accuracy using data from context description grammar 112,user profiles 110, and the agents 106, among other components. At thesame time, the fuzzy set possibilities or prior probabilities for thewords in the context description grammar may be dynamically updated tomaximize the probability of correct recognition at each stage of thedialog. The probabilities or possibilities may be dynamically updatedbased on a number of criteria including the application domain, thequestions or commands, contexts, the user profile and preferences, userdialog history, the recognizer dictionary and phrase tables, and wordspellings, among other criteria.

For uncommon words or new vocabulary words, the user may be given theoption to spell the words. The spelling can be done by saying the namesor the letters or using a phonetic alphabet. The phonetic alphabet canbe a default one or one of the user's choosing.

Alternatively, when a user submits a word that is not recognized at allor is not correctly recognized by the'speech recognition engine 120,then the user may be asked to spell the word. The speech recognitionengine 120 determines this condition based on confidence level for thescoring process. The word may be looked up in a dictionary and thepronunciation for the word is added to either the dictionary, the agent106, or the user's profile 110. The word pronunciation can then beassociated with the domain, the question, the context and the user.Through this process, the speech recognition engine learns with time andimproves in accuracy. To assist users in spelling words, anindividualized phonetic alphabet can be used. Each user can modify thestandard phonetic alphabets with words, which they can remember moreeasily.

Once the words and phrases have been recognized by the speechrecognition engine 120, the tokens and user identification may be passedto the parser 118. The parser examines the tokens for the questions orcommands, context and criteria. The parser may determine a context foran utterance by applying prior probabilities or fuzzy possibilities tokeyword matching, user profile 110, dialog history, and context stackcontents. The context of a question or command may determine the domainand, thereby, the domain agent 156, if any, to be invoked. For example,a question with the keywords “temperature” implies a context value ofweather for the question. Within a different dialog, the keyword“temperature” can imply a context for a measurement. The parserdynamically receives keyword and associated prior probability or fuzzypossibility updates from the system agent 150 or an already activedomain agent 156. Based on these probabilities or possibilities thepossible contexts are scored and the top one or few are used for furtherprocessing.

The parser 118 uses a scoring system to determine the most likelycontext or domain for a user's question and/or command. The score isdetermined from weighing a number of factors including, the user profile110, the domain agent's data content and previous context. Based on thisscoring, the system 90 invokes the correct agent. If the confidencelevel of the score is not high enough to ensure a reliable response, thesystem 90 may ask the user to verify whether the question and/or commandis correctly understood.

In general, the question that is asked by the system 90 may be phrasedto indicate the context of the question including all criteria orparameters. For example, the question can be in the form of: “Did Iunderstand that you want such-and-such” If the user confirms that thequestion is correct the system proceeds to produce a response.Otherwise, the user can rephrase the original question, perhaps addingadditional information to remove ambiguity, or the system can ask one ormore questions to attempt to resolve the ambiguity.

Once the context for the question or command has been determined, theparser 118 can invoke the correct agent 156, 150. To formulate aquestion or command in the regular grammar used by agents, the parserwill preferably determine required and optional values for the criteriaor parameters. These criteria may have been explicitly supplied by theuser or may need to be inferred. The parser may make use of the criteriahandlers 152 supplied by the system agent. The criteria handlers canprovide context sensitive procedures for extracting the criteria orparameters from the user's question or command. Some criteria may bedetermined by executing algorithms in the agent, while others may bedetermined by applying probabilistic or fuzzy reasoning to tables ofpossible values. Prior probabilities or fuzzy possibilities andassociated values may be received from a number of sources including,for example, the history of the dialog, the user profile 110, and theagent. Based on user responses, the prior probabilities or fuzzypossibilities may be updated as the system learns the desired behavior.For a weather context, examples of criteria include, location, date andtime. Other criteria can include command criteria (i.e., yes/no, on/off,pause, stop), and spelling. Special criteria handlers are available fromthe system agent for processing lists, tables, barge-in commands, longstrings of text and system commands.

The criteria handlers 152 can operate iteratively or recursively on thecriteria extracted to eliminate ambiguity. This processing may helpreduce the ambiguity in the user's question or command. For example, ifthe user has a place name (or other proper noun) in their utterance theparser 118 can use services of the domain agent 156 to look up tables inthe databases 102 for place names or can attempt to determine which wordis the proper noun from the syntax of the utterance. In another example,the user asks, “what about flight one hundred and twenty too?” Theparser and domain agent use flight information in the database andnetwork information along with context to determine the most plausibleinterpretation among; flight 100 and flight 20 also, flight 100 andflight 22, flight 122, etc.

Once the context and the criteria are determined, the parser 118 mayform the question or command in a standard format or hierarchical datastructure used for processing by the agents 150, 156. The parser 118 mayfill in all required and some optional tokens for the grammar of thecontext. Often the tokens must be transformed to values and formsacceptable to the agents. The parser obtains the requiredtransformations from the agents, dialog history or user profile 110.Examples of transformations or substitutions performed by the parser ontokens include: 1) substituting a stock symbol for a company name orabbreviation; 2) substituting a numerical value for a word or words; 3)adding a zip code to an address; and, 4) changing a place or other nameto a commonly used standard abbreviation.

The agents 150, 156 may receive a command or question once the parser118 has placed it in the required standard format. Based on the context,the parser can evoke the correct agent to process the question orcommand.

Commands can be directed to the system or to an external entity. Systemcommands are generally directed to the system agent 150. Commands forexternal entities are generally processed by a domain agent 156, whichincludes the command context and behavior for the external entity.

Specific questions may be generally directed to one of the domain agents156. The real-time selection of the correct agent allows the inventionto dynamically switch contexts. Based on the question, command orcontext and the parameters or criteria, the domain agent may create oneor more queries to one or more local or external information sources.Questions can be objective or subjective in nature. Results forobjective questions can often be obtained by structured queries to oneor more local or network information sources. Even for objectivequestions, the system may need to apply probabilistic or fuzzy setanalysis to deal with cases of conflicting information or incompleteinformation. Information to answer subjective questions is generallyobtained by one or more ad-hoc queries to local or network data sources,followed by probabilistic or fuzzy set evaluation of the one results todetermine a best answer.

Once the domain agent 156 has formulated the one or more queries, theymay be sent to local and/or network information sources. The queries maybe performed in an asynchronous manner to account for the fact thatsources respond at different speeds or may fail to respond at all.Duplicate queries can be sent to different information sources to ensurethat at least one source responds with a useful result in a timelymanner. Further, if multiple results are received in a timely manner,they can be scored by the system to determine which data is mostreliable or appropriate. Examples of data sources accommodated include,HTTP data sources, sources with meta-data in various formats includingXML, measurement data from sensors using various formats, device 32setting parameters, entertainment audio, video and game files includingMP3, databases using query languages and structured responses such asSQL, and other data sources.

The local information sources can be stored in one or more systemdatabases 102 or can be on any local data storage such as a set of CDsor DVDs in a player or other local data storage. In other cases, localinformation can be obtained from vehicle system settings or measurementdevices. Network information sources can be connected to, the controland device interfaces 30, the data interfaces 26, the Internet 42 orother network and accessed through a series of plug-ins or adaptors,known a pluggable sources, in the network interface 116. The pluggablesources are capable of executing the protocols and interpreting the dataformats for the data sources of interest. The pluggable source providesinformation scrapping forms and procedures for each source to the domainagents 156. If a new type of data source is to be used a new plug-in oradaptor can be added to the appropriate interface.

The domain agent 156 can evaluate the results of the one or more queriesas they arrive. The domain agent may score the relevance of the resultsbased on results already received, the context, the criteria, thehistory of the dialog, the user profile 110 and domain specificinformation using probabilistic or fuzzy scoring techniques. Part of thedialog history is maintained in a context stack. The weight of eachcontext for the scoring may be based on the relevance of one context toanother and the age of the contexts. Other scoring variables can beassociated through the context stack. Contexts can also be exclusive, sothat previous contexts have no weight in the scoring.

Based on the on-going scoring processes, the domain agent 156 maydetermine if a single best answer can be extracted. For most questions,the desired result may include a set of tokens that may be found toformulate an answer. Once a value has been found for each of thesetokens, the results are ready for presentation to the user. For example,for a question on weather, the tokens can include the date, day of week,predicted high temperature, predicted low temperature, chance ofprecipitation, expected cloud cover, expected type of precipitation andother tokens. Results processed in this manner may include errormessages. For subjective questions, this determination is made bydetermining a most likely answer or answers, extracted by matching ofthe results received. If no satisfactory answer can be inferred from theresults of the query, the agent can do one of the following:

-   -   1. Ask the user for more information, typically through the        speech interface, and based on the results obtained formulate        new queries. This approach is applied when an irresolvable        ambiguity arises in the formulation of a response.    -   2. Formulate new queries based on the results received from the        first set of queries. This approach is typically applied in        cases where the responses received do not include all the        required information. Information sources to queries can be        inferred from the results already obtained (i.e., links in an        HTML document or measurements or settings from other devices 32)        or from other sources. Using this approach one or more sets of        queries and responses can be chained without the need for action        by the user.    -   3. Wait for additional queries to return results.

In any case, the domain agent 156 may continue to make queries andevaluate results until a satisfactory response is constructed. In doingso, the agent can start several overlapping query paths or threads ofinquiry, typically mediated by the event manager 100. This technique,combined with the use of asynchronous queries from multiple data sourcesprovides the real-time response performance required for a naturalinteraction with the user.

The domain agent 156 may apply conditional scraping operations to eachquery response as it is received. The conditional scraping actions maydepend on the context, the criteria, user profile 110, and domain agentcoding and data. For each token to be extracted a scraping criteria 152can be created using the services of the system agent 150. The scrapingcriteria may use format specific scraping methods including, tables,lists, text, and other methods. One or more scraping criteria can beapplied to a page or results set. Once additional results are received,the domain agent can create new scraping criteria to apply to resultsalready acquired. The conditional scarping process removes extraneousinformation, such as graphics, which need not be further processed orstored, improving system performance.

Specific commands are generally directed to one of the domain agents156. The real-time selection of the correct agent allows the inventionto dynamically switch contexts. Command oriented domain agents 156evaluate the command and the state of vehicle systems, systemcapabilities, and measurements to determine if the command can beexecuted at all or if doing so will exceed operating or safety limits.If the command is ambiguous or cannot be executed for some other reason,the system may ask the user for more information or may suggest what theproblem is and a likely approach to the solution. The domain agent mayformat the command for the specific device 32 and control and deviceinterface 30. This formatting may involve variable substitution,inference of missing values and other formatting. Variable substitutionand inference depends on the command context, the user profile 110,command history, state of vehicle systems and measured values, and otherfactors. A complex command can result in more atomic commands being sentto multiple devices, perhaps in a sequence. The sequence and nature ofsubsequent commands may depend on the previous commands, results ofpervious commands, device settings and other measurements. As a commandis executed, measurements are made and results collected to determine ifthe execution was correct and the desired state or states were reached.

Once the domain agent 156 has created a satisfactory response to aquestion, or to a command, the agent may format that response forpresentation. Typically, the domain agent can format the response intothe markup format used by the text to speech engine 124. The domainagent may format the result presentation using available formattemplates and based on the context, the criteria, and the user profile110. The domain agent may perform variable substitutions andtransformations to produce a response best understood and most naturalto the user. The domain agent may vary the order of presentation oftokens and the exact terminology used to create a more natural responseto the user. The domain agent may also select the presentationpersonality 108 to be used.

For both command and query responses, the domain agent 156 may selectthe presentation template, determine order of presentation for tokensand determine variable substitutions and transformations usingprobabilistic or fuzzy set decision methods. The template used to formthe presentation can be from the domain agent itself or from the userprofile 110. The user profile can completely specify the presentationformat or can be used to select and then modify an existing presentationformat. Selection and formatting of presentation template can alsodepend on the presentation personality 108. At the same time, thecharacteristics of the personality used for the response are dynamicallydetermined using probabilities or fuzzy possibilities derived from thecontext, the criteria, the domain agent itself and the user profile 110.

The domain agent 156 may apply a number of transformations to the tokensbefore presentation to the user. These variable substitutions andtransformations may be derived from a number of sources including,domain information carried by the agent, the context, the token values,the criteria, the personality 108 to be used, and the user profile 110.Examples of variable substitutions and transformations include: 1)substitution of words for numbers; 2) substitution of names for acronymsor symbols (i.e., trading symbols); 3) use of formatting informationderived from the information sources (i.e., HTML tags); 4) nature of theresponse including, text, long text, list, table; 5) possible missinginformation or errors; 6) units for measurement (i.e., English ormetric); and, 7) preferred terminology from the user profile orpresentation personality 108.

The invention may provide special purpose presentation capabilities forlong text strings, tables, lists and other large results sets. Domainagents 156 may use special formatting templates for such results. Thesystem agent 150 can provide special criteria handlers 152 forpresentation and user commands for large results sets. The presentationtemplates used by the domain agents for large results sets typicallyinclude methods for summarizing the results and then allowing the userto query the result in more detail. For example, initially only shortsummaries, such as headlines or key numbers, are presented. The user canthen query the results set further. The criteria handlers provide userswith the capability to browse large results sets. Commands provided bythe criteria handlers for large results sets include, stop, pause, skip,rewind, start, and forward.

Some information, in formats such as video, pictures and graphics, maybe best presented in a displayed format. The domain agents 156 applysuitable presentation templates in these cases and present theinformation through the non-speech interface 114. The system agent 150provides special criteria handlers 152 for presentation and usercommands for display presentation and control.

Although particular embodiments of the invention have been shown anddescribed, it will be understood that it is not intended to limit theinvention to the embodiments that are disclosed and it will be obviousto those skilled in the art that various changes and modifications maybe made without departing from the spirit and scope of the invention.Thus, the invention is intended to cover alternatives, modifications,and equivalents, which may be included within the spirit and scope ofthe invention as defined by the claims.

1. A mobile device for processing speech and non-speech communications,comprising: an identification module that identifies a user thatcommunicates with the mobile device; a matching module that matches theidentified user to one or more profiles; a transcription module thattranscribes the speech and the non-speech communications to create aspeech-based textual message and a non-speech-based textual message,wherein the transcription module accesses the one or more profiles; amerging module that merges the speech-based textual message and thenon-speech-based textual message to generate a query; a search modulethat searches the query for text combinations; a comparison module thatcompares the text combinations to entries in a context descriptiongrammar; a plurality of domain agents that are associated with thecontext description grammar; a scoring module that provides relevancescores based on results from the comparison module; a domain agentselector that selects domain agents based on results from the scoringmodule; and a response generating module that communicates with theselected domain agents to obtain content that is gathered by theselected domain agents and that generates a response from the content,wherein the content is arranged in a selected order based on resultsfrom the scoring module.
 2. The mobile device according to claim 1,wherein the response generating module generates an aggregate responsethat includes the content gathered by the selected domain agents.
 3. Themobile device according to claim 1, wherein the identification modulesupports multiple users that access the mobile device during a samesession.
 4. The mobile device according to claim 3, wherein the sessionmay include an overlapping session or an interleaved session.
 5. Themobile device according to claim 3, wherein the identification moduleverifies an identify of the multiple users based on voiceprint matchingor password matching.
 6. The mobile device according to claim 4,wherein, during the interleaved session, the mobile device processes oneor more queries in an order of receipt.
 7. The mobile device accordingto claim 1, wherein, during the interleaved session, the mobile deviceprocesses one or more queries based on an order determined by a lengthof the queries.
 8. The mobile device according to claim 1, wherein themobile device includes (i) a personal digital assistant, (ii) a cellulartelephone, (iii) a portable computer, (iv) a desktop computer, or anycombination of (i) to (iv).
 9. The mobile device according to claim 1,wherein the mobile device receives follow-up speech and non-speechcommunications and wherein the transcription module transcribes thefollow-up speech and non-speech communications to create a follow-upspeech-based textual message and a follow-up non-speech-based textualmessage.
 10. The mobile device according to claim 9, wherein the mergingmodule merges the follow-up speech-based textual message and thefollow-up non-speech-based textual message to generate a follow-upquery.
 11. The mobile device according to claim 1, further comprising apersonality module that facilitates formatting the response.
 12. Themobile device according to claim 1, wherein the mobile device isconfigured to communicate with a context stack that includes one or morecontexts that are selected based on the query.
 13. The mobile deviceaccording to claim 12, wherein the scoring module determines the one ormore contexts based on at least applying prior probabilities or fuzzypossibilities to (i) keyword matching, (ii) user profiles, (iii) adialog history, or any combination of (i) to (iii).
 14. The mobiledevice according to claim 1, wherein at least one of the domain agentscreates and directs a request to a local information source or a networkinformation source.
 15. The mobile device according to claim 1, whereinat least one of the domain agents creates and directs a command to aremote or local device.
 16. The mobile device according to claim 10,wherein the follow-up query is associated with a same context as thequery.
 17. The mobile device according to claim 1, wherein the mobiledevice supports Internet browsing and wherein the generated query isused to conduct the Internet browsing.
 18. The mobile device accordingto claim 1, wherein the mobile device supports multimodalcommunications.
 19. The mobile device, comprising: an object that isaccessible by the mobile device; an input device that receives speechand non-speech annotations corresponding to the object; a transcriptionmodule that transcribes the speech and the non-speech annotations tocreate a speech-based textual message and a non-speech-based textualmessage; and an association module that associates the object with thespeech-based textual message and the non-speech-based textual message.20. The mobile device according to claim 19, further comprising: aclassification module that classifies the speech-based textual messageand the non-speech-based textual message; and a searching module thatsearches the speech-based textual message and the non-speech-basedtextual message based on a search request.
 21. The mobile deviceaccording to claim 19, wherein the association module associates theobject with the speech and non-speech annotations.
 22. The mobile deviceaccording to claim 21, further comprising: a classification module thatclassifies the speech and non-speech annotations; and a searching modulethat searches the speech and non-speech annotations based on a searchrequest.
 23. The mobile device according to claim 19, wherein the objectincludes digital media.
 24. The mobile device according to claim 23,wherein the object includes a digital photograph, a calendar entry, anemail message, an instant message, a phonebook entry, a voicemail entryor a digital movie.
 25. The mobile device according to claim 19, whereinthe association module associates the object to the speech andnon-speech annotations, the speech-based textual message, and thenon-speech-based textual message.
 26. The mobile device according toclaim 25, wherein the object is retrievable using (i) a voice command,(i) a text command, or a combination of (i) and (ii).
 27. The mobiledevice according to claim 19, wherein the object is stored (i) in themobile device, (ii) at a server, or a combination of (i) and (ii). 28.The mobile device according to claim 19, wherein the object includesmetadata.
 29. The mobile device according to claim 28, wherein themetadata includes global positioning system information, environmentalinformation, or geographic information.
 30. The mobile device accordingto claim 28, further comprising a metadata searching module that enablessearching of the metadata.
 31. A system, comprising: a plurality ofmobile devices that support multimodal communications, wherein two ormore mobile devices are associated with a user; and a context modulethat is coupled to the plurality of mobile devices, wherein a samecontext is maintained at the two or more mobile devices that areassociated with the user.
 32. The system according to claim 31, whereinthe plurality of mobile devices comprise: an identification module thatidentifies a user that communicates with the mobile device; a matchingmodule that matches the identified user to one or more profiles; atranscription module that transcribes the speech and the non-speechcommunications to create a speech-based textual message and anon-speech-based textual message, wherein the transcription moduleaccesses the one or more profiles; a merging module that merges thespeech-based textual message and the non-speech-based textual message togenerate a query; a search module that searches the query for textcombinations; a comparison module that compares the text combinations toentries in a context description grammar; a plurality of domain agentsthat are associated with the context description grammar; a scoringmodule that provides relevance scores based on results from thecomparison module; a domain agent selector that selects domain agentsbased on results from the scoring module; and a response generatingmodule that communicates with the selected domain agents to obtaincontent that is gathered by the selected domain agents and thatgenerates a response from the content, wherein the content is arrangedin a selected order based on results from the scoring module.
 33. Thesystem according to claim 31, wherein the context module is configuredto broadcast the same information to two or more mobile devices and toreceive a response from one of the mobile devices.
 34. The systemaccording to claim 31, wherein the context module broadcasts a firsttype of information and receives a second type of information.
 35. Thesystem according to claim 34, wherein first type of information and thesecond type of information include speech information or non-speechinformation.
 36. The system according to claim 35, wherein the firsttype of information includes non-speech information that is presented ona graphical user interface and the second type of information includesspeech information that is applied to search a network.
 37. The systemaccording to claim 36, wherein the first type of information and thesecond type of information correspond to the same context.
 38. A methodof processing speech and non-speech communications in a multimodalenvironment, comprising: receiving the speech and non-speechcommunications; identifying a user that provides the speech andnon-speech communications; matching the identified user to one or moreprofiles; transcribing the speech and non-speech communications tocreate a speech-based textual message and a non-speech-based textualmessage, wherein the transcribing includes accessing the one or moreprofiles; merging the speech-based textual message and thenon-speech-based textual message to generate a query; searching thequery for text combinations; comparing the text combinations to entriesin a context description grammar; accessing a plurality of domain agentsthat are associated with the context description grammar; generating arelevance score based on results from comparing the text combinations toentries in the context description grammar; selecting one or more domainagents based on results from the relevance score; obtaining content thatis gathered by the selected domain agents; and generating a responsefrom the content, wherein the content is arranged in a selected orderbased on results from the relevance score.
 39. The method according toclaim 38, further comprising generating an aggregate response thatincludes the content that is gathered by the selected domain agents. 40.The method according to claim 38, further comprising: receiving afollow-up speech and non-speech communications; transcribing thefollow-up speech and non-speech communications to create a follow-upspeech-based textual message and a follow-up non-speech-based textualmessage; and merging the follow-up speech-based textual message and thefollow-up non-speech-based textual message to generate a follow-upquery.
 41. The method according to claim 38, further comprising apersonality module that communicates the response to a user.
 42. Themethod according to claim 38, wherein the system supports multiple usersduring a same session.
 43. The method according to claim 42, wherein thesession may include an overlapping session or an interleaved session.44. The method according to claim 42, wherein the multiple users areidentified based on voiceprint matching or password matching.
 45. Themethod according to claim 43, further comprising processing one or morequeries in an order of receipt.
 46. The method according to claim 43,further comprising processing one or more queries based on an orderdetermined by a length of the queries.
 47. The method according to claim38, further comprising generating a context stack that includes one ormore contexts that are selected based on the query.
 48. The methodaccording to claim 47, wherein the one or more contexts are generatedbased on applying prior probabilities or fuzzy possibilities to (i)keyword matching, (ii) user profiles, (iii) a dialog history, or anycombination of (i) to (iii).
 49. The method according to claim 38,further comprising conducting Internet browsing with the generatedquery.